Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

141–150 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#141

Post by jakevoytko »

jeffbee wrote: post #81
robswc wrote: post #43

It was if I was in two realities the other day. I did experience a few bugs but twitter never went "down" for me, despite everyone claiming that 1) it would and 2) it already did. All 10 of the "top trends" for me were about how twitter was dead/dying and armchair experts on reddit said it was only a matter of hours at this point. Maybe it was close (in reference to external factors not implied in your very insightfu…

A few parts clearly did go down. 2FA login was just serving error codes all day a few days ago. On Saturday people were posting entire feature films, 2 minutes per tweet, because apparently their copyright content matching wasn’t working.

At least one high-follower account on my timeline posted that their newest notifications were from October

Re: Why Twitter didn’t go down: From a real Twitter SRE

#142

Post by brookst »

porknubbins wrote: post #69
emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivat…

> all software goes down and requires specialist intervention eventually

Well, that’s it, isn’t it? How many software systems need to keep running for Twitter to remain more or less functional?

If there are 10 critical systems that are running at four 9’s, you’d expect 3.6 hours of downtime a year, or about 90 days of uptime at a stretch if I have my math right.

If there are 100 critical systems running at 3 9’s, you’d expect 2.5 hours of downtime per day.

So yeah, all software should keep running. But it doesn’t. And something like Twitter isn’t “a software”, it’s a very large assembly of lots of different software systems and the exponential math that dependencies create.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#143

Post by mradek »

onionisafruit wrote: post #120
dcow wrote: post #110

Earlier quoted context omitted.

I have a piece of Rust software that has not gone down in its entire lifetime.

Me too. It’s a hello world web service I wrote while reading a tutorial after work this afternoon.

Thanks for the laugh I needed that

Re: Why Twitter didn’t go down: From a real Twitter SRE

#144

Post by helloharvey »

MuffinFlavored wrote: post #131
swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

> Twitter grew 3x on the headcount front There were multiple executives making $10m/yr+ There were board members There were shareholders Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"? These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc. How can us commenters on HackerNews sit from our arm…

No post body was provided.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#146

Post by whatshisface »

mikeweiss wrote: post #135
hn_throwaway_99 wrote: post #123

Earlier quoted context omitted.

I did see a lot of politically-motivated salivating which I thought was woefully premature. That said, I see 2 outcomes now: 1. Generally, with large, complex systems like this, everything works, until it doesn't. All the big boys have major outages periodically. I just can't fathom how Twitter is going to handle the eventual certainty of a major outage when, as the author notes, in some cases there are teams that ha…

Wait Musk saddled Twitter with debt? Please elaborate...

It was a leveraged buyout. They are a deal where the company borrows money and combines it with the investor's money to buy all of the shares back.

https://www.washingtonpost.com/business/elon-musks-twitter-d...

Re: Why Twitter didn’t go down: From a real Twitter SRE

#147

Post by user_ »

acdha wrote: post #95
user_ wrote: post #79

Earlier quoted context omitted.

> Twitter added 1.6M daily active users this past week, another all-time high https://twitter.com/elonmusk/status/1594865247323660290

Want to bet that’s not unconnected to sacking the people who tried to block bots?

I'm sure it's connected, but not because of the bots.

Millions of political moderates are stopping by to see what happens when a large-scale public forum begins to tolerate freedom of speech.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#148

Post by throwaway892238 »

devoutsalsa wrote: post #7
cloudking wrote: post #4

Earlier quoted context omitted.

Nice work! Does Twitter runs their own DCs or hosted somewhere else?

As of three years ago, Twitter had 2 to 3 data centers, and was moving some stuff to GCP. Not sure if current state of things.

> and was moving some stuff to GCP

my condolences

Re: Why Twitter didn’t go down: From a real Twitter SRE

#149

Post by hairofadog »

throwawaylinux wrote: post #109
hairofadog wrote: post #105

Regardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is: https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc... ) If this is the new town square, you can forward my mail to a cabin in the woods.

I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.

It doesn’t bother me because it’s lascivious, it bothers me because it’s the comedy equivalent of fart noises. It was one thing when he was just a user of Twitter, but now he’s in charge, and it seems like a tone is being set that’s not really the vibe I’m into. Whatever floats your boat, I guess.
Post reply on HN