Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

931–940 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#931

Post by okdood64 »

mapme wrote: post #298
csomar wrote: post #281

Did the OP confirm Elon theory that most of the stuff is not needed? > For four of those years I was the sole SRE for the Cache team. There was a few before me, and the whole team I worked with, where a bunch came and went. But for four years I was the one responsible for automation, reliability and operations in the team. I designed and implemented most of the tools that are keeping it running so I think I’m qualifi…

One SRE, many SWE. Also have fun asking someone to be permanently oncall with one person on the team. The cache clusters size are also described here for anyone who wants a good technical read over speculation. https://www.usenix.org/system/files/osdi20-yang.pdf

SWEs can share the oncall rotation with SRE.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#932

Post by MockObject »

stonemetal12 wrote: post #925
holoduke wrote: post #506

Earlier quoted context omitted.

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

Googling "how many people work at stack overflow" gives a number less than 400. If true I think you could put together and run Twitter with less than 400 people from an engineering stand point. However Twitter probably has a much higher soft skills head count for moderation and what not. Though it wouldn't surprise me if a lot of that was contractors not direct head count.

I don't see any comparison at all. Does Stack have to maintain mobile apps? Their moderation is mostly handled by users, but Twitter must write and maintain tools for handling that. I doubt anyone is trying to upload illegal porn to Stack. Latency of 30 minutes would be noticed by very few Stack users. Twitter probably really needs at least 10x the SREs of Stack.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#933

Post by hn2017 »

weinzierl wrote: post #464
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

No matter if it was or not and for better or worse: If Twitter survives this without any major harm it will have profound consequences for the whole software industry.

It will be destructive to society in the amount of toxic speech. Lay off all the content moderators, ok buddy..

https://www.theguardian.com/technology/2022/nov/20/twitter-f...

Re: Why Twitter didn’t go down: From a real Twitter SRE

#934

Post by mapme »

okdood64 wrote: post #931
mapme wrote: post #298

Earlier quoted context omitted.

One SRE, many SWE. Also have fun asking someone to be permanently oncall with one person on the team. The cache clusters size are also described here for anyone who wants a good technical read over speculation. https://www.usenix.org/system/files/osdi20-yang.pdf

SWEs can share the oncall rotation with SRE.

Yes that is the normal case. The post was refuting the assertion that one engineer can run these services indefinitely as previously the OP had the help of SWEs oncall and also fixing bugs.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#935

Post by mapme »

csomar wrote: post #328
mapme wrote: post #298

Earlier quoted context omitted.

One SRE, many SWE. Also have fun asking someone to be permanently oncall with one person on the team. The cache clusters size are also described here for anyone who wants a good technical read over speculation. https://www.usenix.org/system/files/osdi20-yang.pdf

The OP claims he did the implementation (so he was the software engineer too?): > I designed and implemented most of the tools that are keeping it running so I think I’m qualified to talk about it.

SRE is a mix of both. The expectation is you are able to write and understand any code the team is responsible for.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#936

Post by WalterBright »

trollerator23 wrote: post #867
WalterBright wrote: post #165

Earlier quoted context omitted.

America is such a great country that a random person can just fecklessly blunder into creating a revolutionary electric car company and cluelessly blunder into creating a rocket company that is the envy of the world.

I _think_ you are joking. No?

I'm ridiculing the popular notion that Musk is the poster boy for "you didn't build that".

Re: Why Twitter didn’t go down: From a real Twitter SRE

#937

Post by matthias509 »

matwood wrote: post #752
itsoktocry wrote: post #676

Earlier quoted context omitted.

> removing engineers won't instantly crash the product. It'll happen slowly It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.

I agree those were odd takes. I've likened firing most of the engineers to taking your hands off the wheel in the car. It won't crash immediately, but it doesn't mean the car can go driverless. With that said, there are differences between internal systems and something like Twitter on the public internet. I assume that Twitter is a system under constant attack. What happens when the next log4shell level vulnerabilit…

I think it’s more like removing captain from a giant ocean going freighter. Everything will be fine till one day it runs aground or hits an iceberg.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#938

Post by btbuildem »

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

> The biggest thing that brings down a site is changes

Absolutely agreed. In that vein, there is such a thing as too much automation. Sometimes, build chains are set up to always pull in the newest and the freshest -- and given the staggering number of dependencies software generally has, this might mean, small changes all the time. Even when your code does not change, it can eventually break.

It's been my experience that a notable part of software development (in the cloud age, anyway) is about keeping up with all the small incremental changes. It takes bodies to keep up with this churn, bodies which twitter now does not have.

It'll be interesting to keep observing this. So far it's been a testament to the teams that built it and set up the infra -- it keeps running, despite a monkey loose in a server room. It's very impressive.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#939

Post by memish »

theclansman wrote: post #494
headsoup wrote: post #53

Earlier quoted context omitted.

Nothing beats an internet random telling us how much a successful person is silly and also can't meet their own superior standards. It's not about being a fan or not, it's that you're not actually providing any real insight other than signalling how smart you are.

These people behave just like the irrational fanboys, except they just do the exact opposite. Being a sheep and being a contrarian sheep are the same thing.

"Knowing that haters are just fanboys with the sign bit flipped makes it much easier to deal with them"

http://paulgraham.com/fh.html

Re: Why Twitter didn’t go down: From a real Twitter SRE

#940

Post by btbuildem »

raldi wrote: post #24

The most helpful thing to reflect on in these Twitter operational discussions is the difference between homeostasis and evolution. You can get rid of 80% of the work force and the existing homeostasis systems will keep things running smoothly despite known day-to-day chaos. Where you’re really going to run into trouble is inventing responses to novel chaos and gradually changing times.

Homeostasis is a good metaphor, but it implies a living, dynamic system. Something that resists entropy by itself being in a state of flow -- the matter constantly changing, while maintaining the form.

In modern software environments, the entropy is almost violent -- the changes in all the constituent dependencies are constant and relentless. Something frozen in time does not stand a chance, unless it's entirely stand-alone and dependency-free -- an unlikely scenario with a service of Twitter's size.

Post reply on HN