Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

671–680 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#671

Post by pas »

bjarneh wrote: post #585
depereo wrote: post #486

Earlier quoted context omitted.

Slack space leads to innovations, like developing infrastructure automation and improving capacity planning. SRE as a practice needs slack space for operations teams to work on improvements and fixes in addition to BAU fault fixing, deployments and patching.

> Slack space leads to innovations I'm certainly no supporter of "lean operations" with minimum staff etc, and fully agree that you need people that are well rested (for the lack of a better analogy) to do great stuff. But I do think that some of these internet giants do have to many people working there; wasn't LinkedIn 14_000 strong when Microsoft bought it? I've always felt that the American model of doing busines…

Well, half of the staff prepares for some conference, meetup, tech talk, or helps organize one, or does 20% time, or sits in unnecessary meetings, or sits in necessary but inefficient meetings, or is on PTO or on unpaid leave in some retreat.

Plus above a certain headcount the communication overhead becomes seriously large, so just to compensate for the lost velocity you need to break out into smaller more agile more autonomous teams, which further increases the coordination requirements (thus the comms overhead), but allows overall throughput to scale.

And the leading edge technologies commonly used require a large headcount to begin with. (I mean just to start running something twitter/linkedin sized requires at least 1 engineer/million users, so a few hundred folks is a given. You need someone who understands networking, from BGP to TLS to VPN to whatever, internal IT, CI/CD, ops/SRE ... at that scale if you use anything, you need an expert for it. You use Kafka with a hundred millions of users? You might need at least a few people who actually know what the fuck a partition means. Unless you want to just directly give all of your money to Jeff in the form of egress fees, you might need folks to setup CDNs, and whatnot.

So without naming names, ina big password manager company (around a hundred million users?) a few years ago there was a certain rewrite project. 3+ people worked on it for 8-10 months, then it was put on hold temporarily. And then obviously nobody speak of it ever.

It happens that there are inefficiencies that for months not one line goes into production from certain individuals.

It was bad management, yes. But if good management was easy to find then we would be talking about different things :)

Re: Why Twitter didn’t go down: From a real Twitter SRE

#672

Post by ehnto »

cm2187 wrote: post #449
twblalock wrote: post #409

Earlier quoted context omitted.

It’s easy to think it’s bloat at a steady state. When something important goes down and nobody knows how to fix it, it looks different.

That being said it's not like twitter is a massively complex product with lots of different features. I can imagine you could keep it running with a skeletton team. Liasing with ads buyers excepted.

I imagine the infrastructure implemented is fairly complex, this post does outline that it's not a super simple operation. It also alludes to there being more business units than just the core application.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#673

Post by bandrami »

drdrey wrote: post #162
throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

Another thing we noticed at Netflix was that after services didn’t get pushed for a while (weeks), performance started degrading because of things like undiscovered memory leaks, threads leaks, disks filling up. You wouldn’t notice during normal operations because of regular autoscaling and code pushes, but code freezes tended to reveal these issues.

Back in the Pleistocene I worked in a ColdFusion shop (USG was all CF back then and we were contractors) and we had two guys whose job was to bounce stacks when performance fell under some defined level.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#674

Post by yunohn »

867-5309 wrote: post #537
yunohn wrote: post #444

Earlier quoted context omitted.

> I mean wholly crap, this reminds me of old German industry where people retire in the place they started. This is honestly uncalled for. Job hopping every 2-3yrs should not be an expected task.

is that down to the employer or employee?

All companies I’ve worked at, startup to FAANG, have tried their best to reduce attrition and retain employees by providing growth opportunities.

Changing jobs is a skill/knowledge net-negative for the employer, and can be negative (outside of salary) for the employee as they have to relearn everything (relationships, tech debt, processes) etc.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#675

Post by Fellshard »

Yoric wrote: post #660
chrsig wrote: post #634

Earlier quoted context omitted.

advertisers wont care in a couple months. the situation has lots of public attention now, which is what the advertisers are actually afraid of. They don't want their brand associated with the craziness. They don't actually care one way or the other about any choice musk makes. They're not going to just walk away from a 300+ million person audience permanently just due to principles

No but if there is no content moderation and Twitter becomes even more of an open cesspit than it already was, announcers are going to stay away because most of them do not want to be associated with toxicity. Perhaps politically active organizations will want to be there (e.g. the NRA or the ACLU), but nothing family-friendly (e.g. Disney) and most likely nothing Christmas-y during Christmas season.

Only if ill-intended parties pressure those advertisers and effectively threaten them with that bad publicity for sticking around. It changes the game when there is coercion by third parties who are more than happy to see Twitter collapse if it isn't governed by their desired policies and worldviews.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#676

Post by itsoktocry »

donkeyd wrote: post #372
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

>removing engineers won't instantly crash the product. It'll happen slowly

It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#677

Post by bombolo »

TranquilMarmot wrote: post #375
drdrey wrote: post #162

Earlier quoted context omitted.

Another thing we noticed at Netflix was that after services didn’t get pushed for a while (weeks), performance started degrading because of things like undiscovered memory leaks, threads leaks, disks filling up. You wouldn’t notice during normal operations because of regular autoscaling and code pushes, but code freezes tended to reveal these issues.

We used to have a horribly written node process that was running in a Mesos cluster (using Marathon). It had a memory leak and would start to fill up memory after about a week of running, depending on what customers were doing and if they were hitting it enough. The solution, rather than investing time in fixing the memory leak, was to add a cron job that would kill/reset the process every three days. This was easier…

USA at some point had an anti-missile system that needed periodical reboots because it was originally designed for short deployments, so the floating point variable for the clock would start to lose precision after a while.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#678

Post by Existenceblinks »

Well, debugging a company in prod mode like this is adventurous. Just like we remove code to see if things break, and put it back piece by piece until it's not. That's how some of us get rid of bloat. (Yes, I'm aware of orange and apple comparison)

Re: Why Twitter didn’t go down: From a real Twitter SRE

#679

Post by x0x0 »

objclxt wrote: post #551
holoduke wrote: post #506

Earlier quoted context omitted.

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

> I am convinced you can run the entire tech stack with a team of a 100 people. This is because you don't see the complexity. What you see as a Twitter user is a fraction of what's actually there. You have to build a platform for ads. Not just serving ads, but allowing advertisers to prepare their collateral, preview them, get their results, and be billed. So that's an entire content and invoicing platform separate f…

Ads isn't just serving, but targeting. The better you target, the more your ads are worth. When you have $5B of advertising, a 0.01% improvement is breakeven for $500k of fully-loaded comp. So you should add as many engineers / data scientists / etc as can generate an 0.01% annual improvement. Or maybe you want to take 3x their annual salary: that's still an 0.03% improvement in ad relevance.

Separately, some commenters here are flatly delusional about the effort to ship a site, android and ios apps, internal mod tools, help docs, support, and legal docs in 34 supported languages. Not to mention obeying laws in all the countries that implies.

Or image and video hosting! With recoding of videos, resizing of images, and the management of what is surely petabytes of images and videos with very high reliability! That is not a 1, 2, or 3 person job to do well.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#680

Post by nashashmi »

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

In addition to the many great comments here, remember that super star engineers don't exactly fix problems from day to day. They fix the problems before they become problems.

The engineer was doing stability planning for 6 months out for the purpose of cost optimization. I guess we can assume that the costs of infrastructure is about to go up and reliability is about to go down in the coming months.

Post reply on HN