Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

61–70 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#61

Post by typon »

userbinator wrote: post #19
danielodievich wrote: post #3

Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.

software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…

Spot on. Absolutely hate this attitude that software sitting there just gathers wear and tear as if it's a mechanical device. Software is written with a particular target platform in mind: x86, ARM, Nvidia GPUs, FPGA soft-processor etc. If the hardware you are running on doesn't change, your software should still function. If the specs of that target platform don't change, your software should still function. If the specs of the target platform change but a hard-working compiler engineer has done the work to make sure your software gracefully uses the new features (for example, a compiler optimizing using AVX instructions), your software should still function.

The fact that most software doesn't continue to function even on the same platform, and on the same hardware, is a massive indictment of the software industry's standard practices.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#62

Post by ReptileMan »

totorovirus wrote: post #44

If Twitter survives and people shut up their mouths complaining after few months we'll see who was right.

And then the other big tech companies will take note. I personally know of highly compensated teams in VMware that do almost nothing.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#63

Post by wankle »

emodendroket wrote: post #57
encryptluks2 wrote: post #50

Earlier quoted context omitted.

Breach of contract possibly. Usually companies don't want you revealing their internal infrastructure publicly unless you get permission. Who knows though.

Wouldn't that be a civil claim and not actually a law being broken then?

There are entire sections of the law around trade secrets (intellectual capital).

Re: Why Twitter didn’t go down: From a real Twitter SRE

#64

Post by headsoup »

etchalon wrote: post #46
taolegal wrote: post #41

Earlier quoted context omitted.

So should the platform be guided by advertisers? Especially one that’s apparently the de facto public square?

I don't think "should" is really a function here. "Must" might be. No one has figured out a way to monetize a social network without advertising, at least not at the scale Twitter has to operate.

$8 blue check marks :p Not sure that has long term profitability...

I think the statement probably is more like "no one has figured out an easier way to monetize a social network"...

Re: Why Twitter didn’t go down: From a real Twitter SRE

#66

Post by emodendroket »

raldi wrote: post #24

The most helpful thing to reflect on in these Twitter operational discussions is the difference between homeostasis and evolution. You can get rid of 80% of the work force and the existing homeostasis systems will keep things running smoothly despite known day-to-day chaos. Where you’re really going to run into trouble is inventing responses to novel chaos and gradually changing times.

I think this is kind of baked in though. Part of the thought process seems to be, at least for non-paying customers, it's not actually necessary to have five nines for Twitter, because people will just put up with it if it's less reliable.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#67

Post by AustinDev »

headsoup wrote: post #42
emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

Because politics and ideologues are so prevalent and everyone's attaching 'the other' (both sides) for the own. And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."

I love this comment. It's so salient. It's probably hard for a lot of people on this board to digest but, I feel like most know where you're coming from.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#68

Post by acdha »

taolegal wrote: post #41
philistine wrote: post #33

Earlier quoted context omitted.

And Twitter's problem are nowhere near technological. The site needed to make more money, not reengineer the whole thing while advertisers are fleeing because Trump is back on on a whim!

So should the platform be guided by advertisers? Especially one that’s apparently the de facto public square?

What choice do they have? The company was hovering around profitable (2021 would’ve been without a lawsuit settlement) but that was before they were saddled with a ton of new debt. It’s possible that they could find new revenue models - that pay for checkmark scheme isn’t it but maybe a smart businessman could come up with a better variant - but that takes time and has to be done carefully since they have to keep the lights on in the middle. Driving away advertisers before finding that new revenue source doesn’t leave much time to iterate.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#69

Post by porknubbins »

emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivations rather than deep knowledge of system failures.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#70

Post by halfmatthalfcat »

swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

This is the real question? Your question has nothing to do with the blog post and if you take a look around, what Twitter did was literally done across the entire industry, hence all the layoffs recently. There was a hiring glut to take advantage of cheap capital during COVID recovery. The capital has dried up, glut has ended and a lot of people lost their jobs. Why is that so hard to see? None of this is unique in any way to Twitter.
Post reply on HN