Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

81–90 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#81

Post by jeffbee »

robswc wrote: post #43

It was if I was in two realities the other day. I did experience a few bugs but twitter never went "down" for me, despite everyone claiming that 1) it would and 2) it already did. All 10 of the "top trends" for me were about how twitter was dead/dying and armchair experts on reddit said it was only a matter of hours at this point. Maybe it was close (in reference to external factors not implied in your very insightfu…

A few parts clearly did go down. 2FA login was just serving error codes all day a few days ago. On Saturday people were posting entire feature films, 2 minutes per tweet, because apparently their copyright content matching wasn’t working.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#82

Post by throwaway0x7E6 »

colesantiago wrote: post #15
danielodievich wrote: post #3

Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.

> Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough. This seems to be more true of Mastodon than Twitter. I can't imagine any self hosted Mastodon instance staying up longer than twitter.

why? a single app running on a single server is several orders of magnitude more resilient than a spaghetti clusterfuck of services upon services. twitter could be brought down by a single expired certificate

Re: Why Twitter didn’t go down: From a real Twitter SRE

#83

Post by TravHatesMe »

a-dub wrote: post #51

i think i prefer a culture where one is rewarded for automating their job. moves like musk's are bad for the industry. they produce more systems with "job security" built in.

> more systems with "job security" built in.

How common is this?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#84

Post by polemic »

userbinator wrote: post #19
danielodievich wrote: post #3

Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.

software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…

Moot in this case. Musk has gutted the work-force and instigated a wide range of changes. All bets are off when new code is being introduced.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#85

Post by stoniejohnson »

blamazon wrote: post #14
irrational wrote: post #12

Very interesting article. But the constant misuse of “to” for “too” kept throwing me out of reading mode. I wonder why my brain does that instead of reading past it since I know what is actually meant.

Interesting, I didn't notice despite reading the article start too finish. The brain is so mysterious!

Irony!

Re: Why Twitter didn’t go down: From a real Twitter SRE

#86

Post by nverno »

ulkesh wrote: post #27

All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…

Judging by some of the old patents he's filed [1], I'd guess he has at least a decent understanding of the tech involved. Probably less so, when it comes to the details of more modern distributed systems, but I also wouldn't be surprised if he's spent some effort towards all that as well - he's been working in/around pretty cutting edge tech for quite a while. Could he sit down and code it himself? probably not, but that's hardly required in his situation.

1. https://patents.justia.com/inventor/elon-musk

Re: Why Twitter didn’t go down: From a real Twitter SRE

#87

Post by poulsbohemian »

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the good times are over." Things start to become brittle and/or not get the attention needed, contractors are brought in because they are cheaper and/or bits get outsourced to the cheapest bidder... the professional care and attention like the author clearly brought just starts to shift over time. Consultants like me are brought in to diagnose what's wrong - the good staff could write our briefs, they know what's going on - and generally we slap a band-aid on the problem because management just wants to squeeze whatever value they can out of the assets rather than actually improve anything.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#88

Post by whatshisface »

wankle wrote: post #59
encryptluks2 wrote: post #50

Earlier quoted context omitted.

Breach of contract possibly. Usually companies don't want you revealing their internal infrastructure publicly unless you get permission. Who knows though.

Yes exactly this. Maybe they did but I didn't find it stated in the writeup, which was actually pretty good. If it were my company though, I'd want the intellectual capital protected and we programmers usually have to sign non-compete agreements so I think my original comment stands, it was my opinion and initial gut reaction, not an indictment or judgement in any way.

Yeah, now that I know that Twitter uses caches and has automatic failover I'll be able to put them out of business by competing with them. :P

Re: Why Twitter didn’t go down: From a real Twitter SRE

#89

Post by rapind »

userbinator wrote: post #19
danielodievich wrote: post #3

Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.

software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…

Don't forget that users have been trained to be incredibly fault tolerant as a result of how flaky general software can be. Now that cars are having BSOD that tolerance may reach new levels or just evaporate.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#90

Post by nemo44x »

headsoup wrote: post #42
emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

Because politics and ideologues are so prevalent and everyone's attaching 'the other' (both sides) for the own. And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."

No post body was provided.
Post reply on HN