Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

211–220 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#211

Post by kweingar »

emodendroket wrote: post #66
raldi wrote: post #24

The most helpful thing to reflect on in these Twitter operational discussions is the difference between homeostasis and evolution. You can get rid of 80% of the work force and the existing homeostasis systems will keep things running smoothly despite known day-to-day chaos. Where you’re really going to run into trouble is inventing responses to novel chaos and gradually changing times.

I think this is kind of baked in though. Part of the thought process seems to be, at least for non-paying customers, it's not actually necessary to have five nines for Twitter, because people will just put up with it if it's less reliable.

I don’t have personal experience in this, so obviously I can’t speak with any authority. But I have heard from colleagues that tons of little factors can dramatically affect user engagement. For example, even a couple dozen milliseconds of longer load times can push a noticeable number of users away from your app.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#212

Post by emodendroket »

hn_throwaway_99 wrote: post #123
emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

I did see a lot of politically-motivated salivating which I thought was woefully premature. That said, I see 2 outcomes now: 1. Generally, with large, complex systems like this, everything works, until it doesn't. All the big boys have major outages periodically. I just can't fathom how Twitter is going to handle the eventual certainty of a major outage when, as the author notes, in some cases there are teams that ha…

Yeah, I find that easy to envision myself, but it's not the kind of spectacular, instant failure that's been predicted at all.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#213

Post by jonahrd »

polio wrote: post #183
dcow wrote: post #110

Earlier quoted context omitted.

I have a piece of Rust software that has not gone down in its entire lifetime.

By definition, no software goes down until its lifetime ends.

I would say the lifetime of a piece of software is different from the lifetime of the process running it.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#215

Post by thakoppno »

jeffbee wrote: post #81
robswc wrote: post #43

It was if I was in two realities the other day. I did experience a few bugs but twitter never went "down" for me, despite everyone claiming that 1) it would and 2) it already did. All 10 of the "top trends" for me were about how twitter was dead/dying and armchair experts on reddit said it was only a matter of hours at this point. Maybe it was close (in reference to external factors not implied in your very insightfu…

A few parts clearly did go down. 2FA login was just serving error codes all day a few days ago. On Saturday people were posting entire feature films, 2 minutes per tweet, because apparently their copyright content matching wasn’t working.

> On Saturday people were posting entire feature films, 2 minutes per tweet, because apparently their copyright content matching wasn’t working.

That’s funny but at the same time it’s easy to find stuff like that on youtube and has been forever.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#217

Post by __warlord__ »

wyclif wrote: post #114
didip wrote: post #93

I am mostly curious about the Mesos layer itself. Mesos is dead. So you need in-house expertise to patch it without being able to leverage community knowledge. Does Twitter retain enough people to manage Mesos?

I was surprised to hear that a cutting edge social media app uses Mesos. Why did they choose that over other options?

IIRC Mesos was an internal tool (at Twiiter) that got released under the Apache umbrella later on.

Then Mesosphere (a company) wanted to bring it to the enterprise market but at the time was competing with Kubernetes... and we all know who won.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#219

Post by chrismarlow9 »

mradek wrote: post #158

Tangent but I do hope that Musks trim down results in orgs that have less “executives” and a layered cake of a org structure, and more autonomous small teams that execute on shared overarching initiatives. I really don’t understand why so many tech companies have like 8 layers of engineering levels. If the argument is that you need more money so more levels, just have a bigger band. Don’t chase titles they don’t mean…

If 10 engineers can build 5 features in a week, then why can't 50 build 25? We easily understand why as engineers on HN, and even managers and investors understand why. But that wont stop them from trying to solve that problem. And some of them do! All of these big FANG companies are as big as they are because at some point they figured out how to solve this problem. But just like time and growth break architectures and software, they also break processes, and they eventually encounter a level where they can't solve it anymore. Then they stagnate, then they either die or solve the new problems. They have to grow to keep investors, and to grow they have to keep investors. That's why it doesn't stop at 12 dudes building an app. But sometimes it does just fine. It's really just a matter of who's investing and what returns they're looking for.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#220

Post by jandrese »

libraryofbabel wrote: post #178
fenomas wrote: post #139

Earlier quoted context omitted.

To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...

Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to. So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building . But eventually people are going to start turning on stoves again… an…

Except that he at the same time demanded that people invent an entirely new dish by the end of the week, and now they are scrambling to try to figure it out. Already the DMCA auto-takedown bot is apparently broken and people are posting entire movies on Twitter. I would expect other peripheral systems to start breaking down as nobody is maintaining them even as other parts of the system are being changed.
Post reply on HN