Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

861–870 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#861

Post by donkeyd »

wikfwikf wrote: post #553
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

data architecture is bloat? Have you implemented a system which stores hundreds of billions of pieces of media content and makes different slices of them immediately available to hundreds of millions of users?

It's bloat to devs who just want to write code. You're kinda confirming the point I was trying to make. Anyone on that list performs an important role at a company of their scale.

I'm currently the first and sole architect on a product that was built by only devs. I really know why I exist.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#862

Post by recuter »

donkeyd wrote: post #846
recuter wrote: post #765

Earlier quoted context omitted.

Quoted post unavailable.

> The job cuts are clearly justified because of the extremely toxic "work culture" I'm very curious in what way the stuff Elon is doing is not creating toxic work culture? >Minimum I'm not denying there may have been toxic work culture before, but I don't see how replacing it with another form of toxic work culture is to be celebrated. Also, writing this: > There is no active independent thought around this subject t…

Up until recently 100% office work was the norm. It still is for most people. Many employers are quite adamant about it, not just Elon.

40 hours per week is also known as the standard work week. A bog standard j-o-b. Those are not reasonable examples of any sort of toxicity.

If these painfully regular employment terms are not competitive with other offers nobody is forced to pick twitter as their employer.

> but I don't see how replacing it with another form of toxic work culture is to be celebrated.

Walk on by. Find a company more suited to your temperament that will give you the pay and perks you think you deserve.

> And then spewing a bunch of vitriol and bile really distracts from any point you were trying to make

> because mommy didn't buy them an xbox

I assure you the child was quite literally stating xbox as the reason for his tantrum. Loudly. Whilst withering on the floor. I'm simply giving an eye witness account.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#863

Post by treis »

randomsearch wrote: post #462
jdminhbg wrote: post #152

Earlier quoted context omitted.

> How can us commenters on HackerNews sit from our armchair and say "ah, goofballs should've just not let headcount get so high!" The cliche HN comment on sites like Twitter (and many, many others, any time headcount comes up) has always been "why do they need so many people?" I've mostly dismissed it the same way I dismiss "I could build Uber in a weekend," but with every other tech giant laying people off, maybe I…

My first job out of Uni was a company that ran financial services, logistic fulfilment, and retail. They had at least a dozen large in-house retail brands, several large warehouses with automation that Amazon was probably behind at the time, and they also ran serviced some of the most well known brands in the country - from their website to customer support to logistics. They had delivery networks, networks of direct…

It is amazing how quickly complexity comes about to serve complexity. Like at $CURRENT_JOB we split everything into a few dozen microservices with their own DBs. Turns out that we need all the data in one place to make some decisions. Shocking I know.

Cue a team of engineers working for a year+ to build a complicated data pipeline to bring all that data back together into a graph DB (not a clue why a graph DB) and build a DSL to write the code to make these decisions.

That's where the engineers go. It's complexity that is introduced setting off a chain reaction of increased complexity to account for the complexity introduced. Then you need complexity to account for that complexity and so on.

Not that it's all self inflicted wounds. Running a site that counts 1/10 of humanity as active users is a hard thing to do

Re: Why Twitter didn’t go down: From a real Twitter SRE

#864

Post by concordDance »

Lutger wrote: post #595
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

I'm puzzled by this statement. Do you think of resiliency as waste? That twitter would have been fine without it? The article makes a point that the reason Twitter is running ok on 20% of personnel at this moment is exactly because it was build to be resilient, not because the personnel was bloated. A large part of this so called bloat, the 80%, was responsible for Twitter to be running right now. Calling this bloat…

How was twitter doing fine a few years back at half the number of employees and same number of users?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#865

Post by 3eto »

karpathy wrote: post #539
memish wrote: post #326

Earlier quoted context omitted.

John Carmack, "Elon is definitely an engineer. He is deeply involved with technical decisions at spacex and Tesla. He doesn’t write code or do CAD today, but he is perfectly capable of doing so." Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon i…

Elon also understands deep neural nets a lot more than I think people imagine. He starts with good intuitions and mental models, but also actively asks for technical deep dives, and has very good retention. E.g. I recall teaching him about our use of focal loss in contrast to binary cross-entropy for the object detection neural net (I said it had given us a 5% bump and he asked to know more) and he understood how it…

Thank you for taking the time to write and share your unique and relevant insight.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#866

Post by nobleach »

itsoktocry wrote: post #676
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

> removing engineers won't instantly crash the product. It'll happen slowly It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.

I'm still on the fence. As an engineering manager, I tend to attach faces to those "jobs". So seeing cuts, I imagine a ton of people that had to go home and tell their friends/family/etc that they no longer had a job.

On the other hand. As an engineer, we tend to attach way too much self importance to our roles. Like if we're not there entering the "numbers" 4 6 15 16 24 32 every 108 minutes, the entire business is going to crumble. So... this is one I'm going to watch with a keen eye.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#867

Post by trollerator23 »

WalterBright wrote: post #165
ulkesh wrote: post #27

All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…

America is such a great country that a random person can just fecklessly blunder into creating a revolutionary electric car company and cluelessly blunder into creating a rocket company that is the envy of the world.

I _think_ you are joking. No?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#868

Post by celim307 »

donkeyd wrote: post #830
throw0101c wrote: post #744

Earlier quoted context omitted.

> In a start-up many of these roles can be ignored, becaus growth > stability. In a large organization, part of the bloat helps insure a certain amount of stability that's necessary to keep an organization alive. It also (a) increases the bus factor, [1] and (b) allows people to take vacations and time off without having to watch their phones like hawk. [1] https://en.wikipedia.org/wiki/Bus_factor

Good point. I know this as the Mack Truck Theory. For the project I'm working on right now, there's a couple of incredibly valuable people that would cause a pretty significant issue if they disappeared.

Sounds like those people are in strong negotiation positions

Re: Why Twitter didn’t go down: From a real Twitter SRE

#869

Post by donkeyd »

Yoric wrote: post #663
holoduke wrote: post #506

Earlier quoted context omitted.

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

Not tech but I'm convinced that just answering LEO requests from all over the world takes at least 500 people. That's without their managers, payrolls, etc. Likewise, bringing in Ad money would be a few more hundreds, because you need to chase leads in all countries. Getting the Ads to work? That's tech and I'd be surprised if it was less than 100 people, too.

For a second I was confused how Starlink requests were relevant for Twitter. Then I realized you meant 'law enforcement officer' not 'low-earth orbit'.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#870

Post by ineedasername »

I wasn't really expecting a large failure quite yet. I'm guessing there are enough people left with secondary or tertiary knowledge of systems to keep things ticking over for now. The question is how long that state of affairs can continue before some combination of worker burnout or deferred maintenance will stretch things too thin. Then, all-hands emergency work to get things running could contribute even more to burnout -> resignations etc.

They could stay ahead of things if they hire into areas related to infra maintenance that got hit the hardest, before issues reach the point of cascading failure. Or maybe put everything not critical to keeping the site up & running on hold as remaining devs get a bit of cross training using the limited people w/ institutional knowledge that remain as trainers.

For myself, I don't see much value in Twitter, in terms of net social value. It's format seems pretty much designed for only the most surface level discussions, part of what I believe leads to some of its toxicity: it's simply too hard to have conversations complex enough to invite enough discourse for that to tip the productive/toxic ratio a bit more positive.

So I've been rooting for Twitter to fail for years. I'm not rooting against Musk: I like his other businesses and at least a portion of their ongoing success is tied to his persona. (At least before the Twitter stuff: His actions there may impact his relationship with major banks that his other businesses will rely on, and Musk & JP Morgan, a bank that wasn't included in the deal & therefore won't be hurt by it, is already on somewhat negative terms with Musk. The twitter deal has added a few more to that list, and other banks have undoubtedly taken note. Some bank will always finance the regular sorts of things any large corporation need for him, but they're all going to be pricing in some additional risk. That's not really a big deal, it's just that I think Musk's persona has previously been a net positive for his companies and now it's lost at least a little bit of that.)

Post reply on HN