Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

301–310 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#301

Post by kaba0 »

8n4vidtmkvmk wrote: post #187
jeffbee wrote: post #81

Earlier quoted context omitted.

A few parts clearly did go down. 2FA login was just serving error codes all day a few days ago. On Saturday people were posting entire feature films, 2 minutes per tweet, because apparently their copyright content matching wasn’t working.

that's surprising actually. I'd expect my sites to run for about a year if i suddenly died. maybe longer. certs will renew themselves, servers will restart themselves. everything is on autopay. the thing to kill it would probably be api breaks by 3rd parties. maybe disk space would kill me eventually too.. i don't have auto resizing for my sql database but its got a long runway yet. or god forbid my site grows on its…

I mean, twitter is incomparably more complex than your website (I assume you don’t sit over google or Facebook), plus I’m sure elon made people poke at the system (“we only need 20% of the mucroservices”…)

Re: Why Twitter didn’t go down: From a real Twitter SRE

#302

Post by pooper »

eru wrote: post #259
hn_throwaway_99 wrote: post #123

Earlier quoted context omitted.

I did see a lot of politically-motivated salivating which I thought was woefully premature. That said, I see 2 outcomes now: 1. Generally, with large, complex systems like this, everything works, until it doesn't. All the big boys have major outages periodically. I just can't fathom how Twitter is going to handle the eventual certainty of a major outage when, as the author notes, in some cases there are teams that ha…

Generally agreed. However keep in mind that bankruptcy is a purely financial event and doesn't necessarily have to have any impact on operations. Airlines are notoriously for going bankrupt regularly.

Airlines get free government money though. Will twitter? If not, Twitter will need to borrow more money and at higher interest rates compared to market rates if they go through with bankruptcy.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#303

Post by hot_gril »

You can usually do more with less if the remaining few are up to the challenge. It will come at some cost, like having to "move fast and break things" more. There can even be a positive side if the previous situation had a lot of friction due to "too many cooks," as I've experienced on several teams. What's more worrisome is the alleged 80hr work weeks. Yes, two SWEs on a team aren't as good as one SWE working double hours, since SWE-time is known not to scale that way. But burning people out, aside from being unethical, is risky when nobody is ready to fill the spot.

Musk has a point, but it feels more like he's doing this out of financial desperation.

I'll bet it starts going down a bit more often, nothing too severe. Bigger issue might be inability to roll out new features, especially if they dig a horrible tech debt hole. Maybe someone has more details, I haven't kept up, but it seems like the $8 verification rollout got botched because they cut corners on actually verifying the subscribers.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#304

Post by emodendroket »

miked85 wrote: post #275
libraryofbabel wrote: post #178

Earlier quoted context omitted.

Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to. So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building . But eventually people are going to start turning on stoves again… an…

> Twitter isn’t shipping many new features right now When was the last major new feature? The site has always seemed pretty stagnant.

They outright built me-too versions of both Clubhouse and IG Stories, and a lot of stuff trying to encourage people to be nice.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#306

Post by harriet1 »

randyrand wrote: post #160
wankle wrote: post #30

I can't help but feel it might not be legal to post all those details.

given that it’s showing Twitter in good light, it would be super silly to pursue anything IMO, even if it were illegal. What would be a realistic fine in terms of damages (in $$)? You could argue its actually negative damages.

[dead]

Re: Why Twitter didn’t go down: From a real Twitter SRE

#307

Post by memish »

throwawaylinux wrote: post #109
hairofadog wrote: post #105

Regardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is: https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc... ) If this is the new town square, you can forward my mail to a cabin in the woods.

I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.

That post really did bring the church ladies out. Puritanism runs deep.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#308

Post by hairofadog »

Operative0198 wrote: post #206
hairofadog wrote: post #105

Regardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is: https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc... ) If this is the new town square, you can forward my mail to a cabin in the woods.

You do know that you can block Elon's account.

No post body was provided.
Post reply on HN