Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

261–270 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#261

Post by EdwardDiego »

afavour wrote: post #224
libraryofbabel wrote: post #178

Earlier quoted context omitted.

Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to. So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building . But eventually people are going to start turning on stoves again… an…

> Twitter isn’t shipping many new features right now Is that true? I thought one of Elon’s big pushes was launching the whole Blue Tick subscription thing. That doesn’t feel like a small feature.

Given that Twitter already offered premium API access, they've got billing in place, so now they add a new form that, once your credit card is verified, flicks a boolean on an account that was previously flicked by another process.

It might not be small, but it's not exactly huge.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#262

Post by clayhacks »

He says at the end his automation allowed him to automate more. A great positive feedback loop. I see the inverse is also true. Where having no automation makes automating new things harder. Without enough engineers I think they will hit some issue and just not have the bandwidth to fix it in a reasonable amount of time

Re: Why Twitter didn’t go down: From a real Twitter SRE

#263

Post by mtejo »

ggm wrote: post #230

The article is good, and informative but a little odd in one respect. It uses air-quotes to introduce concepts like "rack" and "shelf" in a DC, but does O(logn) notation mid-flow. If you don't know what a rack is, how are you meant to know what the Order of scaling function means? Thats a highly computer sciences specific notation, and if you grok O(n) you know what a rack, a host, a DC is.

Hah fair point, I was trying to write for a more general audience. Part of blogging for me is to improve my written communication skills. Appreciate this!

Re: Why Twitter didn’t go down: From a real Twitter SRE

#264

Post by eru »

AtlasBarfed wrote: post #172
mikeweiss wrote: post #135

Earlier quoted context omitted.

Wait Musk saddled Twitter with debt? Please elaborate...

Yeah, it's private equity 101. I can't believe it happens. A bunch of MBA sharks get loans to buy something with some chop shop plan or other ruthless scheme to carve up or spitshine a company. The loans are then assigned to the company they bought, rather than the core sharks ... uh investors. The "investors" then repackage/selloff/spit shine as necessary to get it resold to some sucker. The investors get THEIR mone…

You make it sound like it's a bad thing.

The people who buy the debt know what they are getting into. (And if those 'suckers' don't know, honestly, they shouldn't be investing in junk bonds etc.)

Re: Why Twitter didn’t go down: From a real Twitter SRE

#265

Post by kaba0 »

jandrese wrote: post #220
libraryofbabel wrote: post #178

Earlier quoted context omitted.

Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to. So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building . But eventually people are going to start turning on stoves again… an…

Except that he at the same time demanded that people invent an entirely new dish by the end of the week, and now they are scrambling to try to figure it out. Already the DMCA auto-takedown bot is apparently broken and people are posting entire movies on Twitter. I would expect other peripheral systems to start breaking down as nobody is maintaining them even as other parts of the system are being changed.

2FA as well

Re: Why Twitter didn’t go down: From a real Twitter SRE

#266

Post by bagels »

foobazgt wrote: post #207
brookst wrote: post #142

Earlier quoted context omitted.

> all software goes down and requires specialist intervention eventually Well, that’s it, isn’t it? How many software systems need to keep running for Twitter to remain more or less functional? If there are 10 critical systems that are running at four 9’s, you’d expect 3.6 hours of downtime a year, or about 90 days of uptime at a stretch if I have my math right. If there are 100 critical systems running at 3 9’s, you…

Yep, and when one of the SEVs rolls around that would have been small (say 5m of downtime fixed with a flag flip), it instead will have a nontrivial chance of escalating into a major multi-hour/multi-day outage without the right institutional knowledge. I'd guesstimate that Twitter probably has dozens of services that are in the critical path of an average user interaction. It's hard to keep even logically optional d…

It has effects on engagement, retention, and ad revenue though.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#267

Post by ggm »

mtejo wrote: post #263
ggm wrote: post #230

The article is good, and informative but a little odd in one respect. It uses air-quotes to introduce concepts like "rack" and "shelf" in a DC, but does O(logn) notation mid-flow. If you don't know what a rack is, how are you meant to know what the Order of scaling function means? Thats a highly computer sciences specific notation, and if you grok O(n) you know what a rack, a host, a DC is.

Hah fair point, I was trying to write for a more general audience. Part of blogging for me is to improve my written communication skills. Appreciate this!

Something like "scales linearly" or "scales exponentially" probably covers it, for whatever case it was (I forget)

really good article btw. really enjoyed reading it.

might pay to flag when systems you talk about are twitter-internal or are open-source. People love that kind of thing. "wow: twitter uses the gnu C compiler" type thing.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#268

Post by fenomas »

jonas21 wrote: post #236
fenomas wrote: post #139

Earlier quoted context omitted.

To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...

Perhaps a closer analogy would be that the new boss rolled in and threw away 80% of the fire extinguishers. Whether that will spell disaster when there's a fire depends on whether the building had too many fire extinguishers to begin with and whether the boss can buy new, better fire extinguishers to replace some of them before there's a fire.

> Perhaps a closer analogy would be that the new boss rolled in and threw away 80% of the fire extinguishers.

If we're deep-diving it'd be closer to say that he rolled in and sold 80% of the stuff, largely sight-unseen, and if a fire breaks out he'll find out how much of that stuff was fire extinguishers.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#269

Post by sangnoir »

porknubbins wrote: post #69
emodendroket wrote: post #23

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivat…

> I think a lot of people just jumped to the second level based on political motivations rather than deep knowledge of system failures.

Anyone who has ever been oncall can intuit how often stuff breaks in big or little ways. Sometimes it's transient and goes away, sometimes it can be filed away to be fixed in the next year, but sometimes, it turns out to be an all-hands-on-deck crisis for a team, or 5.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#270

Post by EdwardDiego »

dcow wrote: post #110
porknubbins wrote: post #69

Earlier quoted context omitted.

The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivat…

I have a piece of Rust software that has not gone down in its entire lifetime.

Good for you!

Now, if your Rust code was a distributed system that handles spiky loads from ~330m users, and processes petabytes of data, then I'd consider your comparison relevant to Twitter.

But I'm going to assume it's not relevant.

P.S., I've written Java services that never went down, because they had a well defined domain and all potential errors were handled. But, I'm not about to compare that to all of frigging Twitter.

Post reply on HN