Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

451–460 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#451

Post by mjr00 »

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

It's crazy to think about, but many people who use and build software today, including HN readers/commenters, are young enough to have only been exposed to the SaaS, cloud-first era, where software built with microservices deployed from CI/CD systems multiple times per day is just the way things are done.

You're totally right; if you don't make changes to the software, it's unlikely to spontaneously stop working, especially after that first 6-12 months of "hardening" where bugs are found and patched.

Many people working in tech have never been exposed to a piece of software which isn't being constantly changed in small increments and forced upon end users. People are assuming that software is inherently unstable simply because they never use anything that isn't a "cloud service".

This probably comes off as "old man yells at cloud" but I'm not trying to bash cloud here. The cloud/SaaS approach has a ton of advantages for both consumers and businesses. But the average tech person in their 20s vastly underestimates how stable software can be when you aren't constantly pushing new features.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#452

Post by tempestn »

28304283409234 wrote: post #419
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Argh. "It works now, so it will work until forever." It takes _effort_ to make it work this smoothly now, _and in the future_. SRE is about _preventing_ issues. Not mopping up after them. To me, the article read like every succesfull sysadmin story: there's no fires, so sysadmin must be bloat.

I think you're misinterpreting the comment you're replying to. They would agree with you that the tiny SRE team described in the article sounds very effective, and likely have a lot to do with why the site is still up and running currently. Work like that should continue. But if 1-3 people can have that degree of impact, what are the other 8000 doing? (Again, this is just me attempting to interpret the point made by the parent, not trying to make one myself.)

Re: Why Twitter didn’t go down: From a real Twitter SRE

#453

Post by fastball »

gitgud wrote: post #433
mirzap wrote: post #402

Earlier quoted context omitted.

Quoted post unavailable.

Quoted post unavailable.

Literally his first two companies, Zip2 (CTO) and x.com (CEO), were pure software companies?

Another company he founded, SpaceX, is the first space launch provider to ever achieve propulsive landing and re-use of an orbital-class rocket, a feat which includes some of the hardest software engineering you can do.

Another company he runs, Tesla, is famous for being one of the few car companies that makes software for their cars that people actually want to use.

If a citation is needed, it is a citation to explain how anyone could possibly believe with that track record that Elon Musk doesn't know how software is built.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#454

Post by Hnrobert42 »

octodog wrote: post #430
henrikschroder wrote: post #406

Earlier quoted context omitted.

Your reply is indistinguishable from a religious person exclaiming that "God is omniscient and moves in mysterious ways!" Stop worshipping billionaires.

This is a ridiculous take and you are not really adding anything to the discussion. Musk has successfully run companies in the past and is running two extremely successful companies right now.

I disagree. GP was pointing out that GGP was deifying Musk in sort of a cult of personality. I didn’t notice that until GP’s post.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#455

Post by pkolaczk »

j3th9n wrote: post #436
aeyes wrote: post #404

Earlier quoted context omitted.

You don't even need that, the kernel OOM killer would take care of this eventually. Unless its something like Java where the garbage collector would begin to burn CPU.

The OOM killer doesn't restart (randomly, unless configured) killed processes, it just kills.

If it's deployed in K8s, it would be restarted automatically after dying.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#456

Post by twelve40 »

cm2187 wrote: post #449
twblalock wrote: post #409

Earlier quoted context omitted.

It’s easy to think it’s bloat at a steady state. When something important goes down and nobody knows how to fix it, it looks different.

That being said it's not like twitter is a massively complex product with lots of different features. I can imagine you could keep it running with a skeletton team. Liasing with ads buyers excepted.

a bunch of people I talk to say it is massively complex but typically fail to explain how, especially given the super-glacial pace at which they added new features for 15 years. And yes, this article kind of doesn't disprove the bloat at all, unless every single SRE quit? but that is also not what's stated in there.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#457

Post by groffee »

ALittleLight wrote: post #381
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

I think Twitter was stuck in a not-especially profitable niche. They shift into fast-mode to get out of it and find a better spot, then they can shift back into stable mode once they occupy a better equilibrium. That said, there are lots of bugs in Twitter now, today, when they presumably had the benefit of being in stable mode for a long time. For example, Twitter regularly refreshes and loads new tweets while I'm r…

That's not a bug, it's a 'feature', tons of sites do it for whatever asinine reason.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#458

Post by hcazz »

octodog wrote: post #430
henrikschroder wrote: post #406

Earlier quoted context omitted.

Your reply is indistinguishable from a religious person exclaiming that "God is omniscient and moves in mysterious ways!" Stop worshipping billionaires.

This is a ridiculous take and you are not really adding anything to the discussion. Musk has successfully run companies in the past and is running two extremely successful companies right now.

To play devil's advocate for a bit, are you certain he's solely responsible for the successes of SpaceX/Tesla, or that those companies succeeded in spite of his leadership?

In case it isn't clear, I agree with the parent comment. In the same tone as the comment above the parent, you could also write "Thing is - Elon Musk knows a thing or two about sexual harassment lawsuits. It's inevitable that he will have another at Twitter."

Stop idolizing billionaires.

>Musk has successfully run companies in the past and is running two extremely successful companies right now.

Is he actually running those companies, or is he posting selfies of whiteboards at 1:30am showing architecture diagrams of Twitter?

In these markets, after he was nearly forced by courts into completing a purchase deal for Twitter after making an overpriced offer, I'm genuinely curious to see how he pays the ~$1b in interest per year on the debt to purchase Twitter.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#459

Post by WA »

ALittleLight wrote: post #381
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

I think Twitter was stuck in a not-especially profitable niche. They shift into fast-mode to get out of it and find a better spot, then they can shift back into stable mode once they occupy a better equilibrium. That said, there are lots of bugs in Twitter now, today, when they presumably had the benefit of being in stable mode for a long time. For example, Twitter regularly refreshes and loads new tweets while I'm r…

They replaced their somewhat productive engineering workforce with completely unproductive interest payments. I’m not convinced that this will lead to a better spot.

But regarding the bugs, I’m totally with you. Same here. I use Twitter only in the browser. Browse long enough and the page reloads as if it ran out of memory.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#460

Post by [deleted] »

donkeyd wrote: post #372
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

[deleted]
Post reply on HN