Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

71–80 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#71

Post by zdragnar »

user_named wrote: post #21
enneff wrote: post #5

Earlier quoted context omitted.

Right. It’s not like if all the engineers walk out it’s suddenly going to fall over. It’s that when an issue comes up there may not be the expertise on hand to fix it. So the remaining twitter engineers should expect a rough time over the coming months.

Small issues going unfixed, building up over time and cascading into a crisis.

You assume that small issues are ever prioritized in the first place. My experience with large tech organizations is that their priories are driven by internal culture, not best practices, just like any other company.

Although I haven't worked at Twitter, if it is anything like the places I have worked at, the small issues get ignored because nobody makes their career on things that don't show up on investor / shareholder / upper management radars. If it can get papered over so it ends up being someone else's problem in the future, what actually gets worked on is flashier things like redesigning the timeline AGAIN so users spend more time on the website or (in the case that caused me to work for small startups again) spend three weeks deciding on a super trivial design decision with 12 stake holders.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#72

Post by headsoup »

emodendroket wrote: post #49
headsoup wrote: post #42

Earlier quoted context omitted.

Because politics and ideologues are so prevalent and everyone's attaching 'the other' (both sides) for the own. And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."

I don't even doubt that they did a lot of work, but as this article suggests, that's part of the reason why we _wouldn't_ expect the site to just go down in a couple days. If all the servers needed constant manual massaging that would speak more poorly to their work than the other way around.

I guess that highlights that for some time we just won't know the actual effects and everything right now is just posturing and speculating.

If in six months (or suitable time period) we see stagnation in releases and technical issues increasing we'll have some data to infer from.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#73

Post by jdminhbg »

userbinator wrote: post #19
danielodievich wrote: post #3

Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.

software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…

Even if the entirety of Twitter.com were mathematically proven correct, it still would run on servers that are made of physical bits that are subject to entropy.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#74

Post by fragmede »

The "why hasn’t the site gone down" stuff is the wrong conversation. The engineers built a train engine, and this robber baron held up the train and forced the engineers off. Inertia will keep the train going for a good long while because they did such a good job of building the train.

At some point the engin’s fluids will run out and need topping off. There's a warning system though so someone'll get notified before the engine blows up, so it's entirely possible this hardcore crew of H1Bs pulls it off. (Great job replacing the SSL cert!) At some point the train’s boiler will need retrofitting. A new team could successfully replace the boiler, but doing it while the engine keeps running just isn't an easy job.

The handful of devs and SREs at Twitter will do their damnedest to keep the train, err, site running. They might even succeed. That doesn’t prove the site was over-engineered, it means they built a damn fine train engined that successfully pulled the whole train up and over the mountain pass.

Twitter wasn't run like a lean start-up type business. Because wasn’t one. Society needs better than that. The vulture capitalism mindset is what's wrong with this county. What's the barest minimum I can pay people to work for me? Not; what can I pay them to let them live prosperous lives but what's the bare, subsistence minimum.

America needs a middle class and these people were part of it. You don’t get a middle class by injudiciously paring things down to a not-even-skeleton crew that’s going to be worked till they burn out, only to get fired.

Twitter was only $14 million from profitability in Q2, and any idiot can randomly fire people to find that much in that large an organization, but cutting costs like that is to misunderstand the situation entirely. Just like starving yourself isn't healthy and won’t improve your self image, randomly cutting costs like that saves money, but doesn’t result in a healthy business.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#75

Post by acdha »

yieldcrv wrote: post #31
acdha wrote: post #16

Earlier quoted context omitted.

The problem is that it’s run by someone who overpaid badly and needs to come up with significantly more money to pay for the debt he saddled the company with, and then his actions seriously disrupted ad revenue. That puts him more in the PE playbook of cutting costs as deeply as possible even at the extent of long-term growth. Unlike his other companies there isn’t strong government support to drive business for Twit…

He's going to sell Tesla shares to pay off his debts, he's knocked off $4 billion already, he'll be able to keep doing that, make offers to buyout stakes of Twitter from people that really don't want to be involved in this shit show. Twitter will operate at lower costs, and also not be profitable for him given current information. That's financially fine.

We’ll see - it’s an expensive hobby and the banks he borrowed from are probably not going to have a great sense of humor.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#76

Post by mongol »

userbinator wrote: post #19
danielodievich wrote: post #3

Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.

software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…

If software ran without side effects, perhaps. But it doesn't. Databases grow, files are uploaded, logs pile on, messages and events propagates and filesystems fill up. This is why entropy matters.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#77

Post by hotpotamus »

etchalon wrote: post #46
taolegal wrote: post #41

Earlier quoted context omitted.

So should the platform be guided by advertisers? Especially one that’s apparently the de facto public square?

I don't think "should" is really a function here. "Must" might be. No one has figured out a way to monetize a social network without advertising, at least not at the scale Twitter has to operate.

I say everyone can join for free and gets one post a day. $8/month plan includes 10 posts a day, and unlimited for $20. While I have no interest in Twitter, I'm told it's quite addictive, so make the first taste free and then reel them in once they're hooked.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#78

Post by zapdrive »

I know most of my comment is going to be redundant, but I need to say it in order to point to it an year or two from now as an "I told you so!"

Twitter is going to be here for a long long time. It's not going to suddenly shut down, it's not going to slowly decline, there is not going to be mass abandonment.

Elon might be cocky, but at the end of the day he is a successful businessman. He didn't just sent out that loyalty pledge email out of cockyness. He really wanted to kick out everyone who doesn't believe in him and his beliefs. And he sent that email well after understanding how Twitter runs and putting his loyal people from Tesla ans SpaceX in charge of operations.

The woke crowd needs to get out of denial and start coping with the fact that Twitter is not the bastion of unopposed woke ideology anymore.

Post reply on HN