Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

501–510 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#501

Post by DharmaPolice »

carabiner wrote: post #477
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Yeah it's dancing around the question: Was Musk right? All signs so far are pointing to, yes . MBA's will be studying this for years.

It's way too soon to tell. On a dramatically smaller scale my team went through a big drop in headcount. Day 1 the impact was nil. Day 10 the impact was negligible. Day 30 some minor problems were identified. But it wasn't until about Day 90 that we had our first outage and Day 270 that we had our first lengthy outage.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#502

Post by enkid »

tempestn wrote: post #452
28304283409234 wrote: post #419

Earlier quoted context omitted.

Argh. "It works now, so it will work until forever." It takes _effort_ to make it work this smoothly now, _and in the future_. SRE is about _preventing_ issues. Not mopping up after them. To me, the article read like every succesfull sysadmin story: there's no fires, so sysadmin must be bloat.

I think you're misinterpreting the comment you're replying to. They would agree with you that the tiny SRE team described in the article sounds very effective, and likely have a lot to do with why the site is still up and running currently. Work like that should continue. But if 1-3 people can have that degree of impact, what are the other 8000 doing? (Again, this is just me attempting to interpret the point made by…

So 1-3 people have a big impact, the other 7997 must not be doing anything? I don't think that logic follows.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#503

Post by dfghbbjiiugb »

mjr00 wrote: post #451
throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

It's crazy to think about, but many people who use and build software today, including HN readers/commenters, are young enough to have only been exposed to the SaaS, cloud-first era, where software built with microservices deployed from CI/CD systems multiple times per day is just the way things are done. You're totally right; if you don't make changes to the software, it's unlikely to spontaneously stop working, esp…

Your broad point is obviously correct (most outages are caused by code or config changes) but there are still classes of failures that can happen without any real changes, like various performance degradations (maybe your table grows too large) or occasional catastrophic failures from things like disk space or id overflow or something.

There's also the stability of third party systems: forced deprecations, security EOL, etc. The cert expiration stuff people have been mentioning is in this category too. I wouldn't be surprised if something does slip through the cracks at Twitter in the next 4 or 6mo.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#504

Post by pcthrowaway »

tempestn wrote: post #452
28304283409234 wrote: post #419

Earlier quoted context omitted.

Argh. "It works now, so it will work until forever." It takes _effort_ to make it work this smoothly now, _and in the future_. SRE is about _preventing_ issues. Not mopping up after them. To me, the article read like every succesfull sysadmin story: there's no fires, so sysadmin must be bloat.

I think you're misinterpreting the comment you're replying to. They would agree with you that the tiny SRE team described in the article sounds very effective, and likely have a lot to do with why the site is still up and running currently. Work like that should continue. But if 1-3 people can have that degree of impact, what are the other 8000 doing? (Again, this is just me attempting to interpret the point made by…

The SRE team mentioned in the article is SRE for one component of a complex architecture.

There are probably many such components; I'd imagine SRE alone would be 200+ people

How many of the remaining staff have the knowledge required to keep all of those components running smoothly?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#505

Post by enkid »

carabiner wrote: post #477
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Yeah it's dancing around the question: Was Musk right? All signs so far are pointing to, yes . MBA's will be studying this for years.

What do you mean by "right"? He grossly overpaid for a business that isn't profitable and likely made it even less profitable. Even if it limps along and some husk of Twitter survives, it's hard to see Musk as making the right move here.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#506

Post by holoduke »

donkeyd wrote: post #372
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#507

Post by jiggawatts »

oxfordmale wrote: post #443
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Let's do a thought experiment and see what functions aren't needed to keep the light on for 30 days: 1) HR 2) Legal 3) Sales 4) Marketing 5) Payroll 6) Admin staff 7) Most of Engineering, other than the bare minimum of L1/L2/3 support. As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend.

This is more like a car with just the bare essentials: an engine, steering wheel, brakes, and the like.

No: radio, air-conditioning, seat padding, wipers, lights, radar, etc...

Oh, and no maintenance.

It'll drive... for now. But that's it.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#508

Post by wraptile »

minhazm wrote: post #373
memish wrote: post #326

Earlier quoted context omitted.

John Carmack, "Elon is definitely an engineer. He is deeply involved with technical decisions at spacex and Tesla. He doesn’t write code or do CAD today, but he is perfectly capable of doing so." Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon i…

Many people who have worked with Musk have shared similar sentiments in interviews. But it seems that people just refuse to believe any of it. People think that there's no way it's possible for someone to be that deeply technical and be a CEO of multiple companies at the same time. I've talked to people about it and they straight up refuse to believe it saying that it's impossible and that any evidence of him being t…

Something doesn't compute in this scenario though. Either his tricking everyone around him or is unfortunate enough to slip up publically. Not knowing what GraphQL is and talking about RPCs in HTTP is a very revealing slip up.

My guess would be that he has some knowledge but also is very good at faking it which is not necessarily a bad thing - those are good traits for a CEO. Though people should be aware of this fact when evaluating the whole persona.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#509

Post by ldng »

fastball wrote: post #453
gitgud wrote: post #433

Earlier quoted context omitted.

Quoted post unavailable.

Literally his first two companies, Zip2 (CTO) and x.com (CEO), were pure software companies? Another company he founded, SpaceX, is the first space launch provider to ever achieve propulsive landing and re-use of an orbital-class rocket, a feat which includes some of the hardest software engineering you can do. Another company he runs, Tesla, is famous for being one of the few car companies that makes software for th…

Can people stop attributing Space success to Musk when it's Gwynne Shotwell's work (with lots of public subsidies)

Re: Why Twitter didn’t go down: From a real Twitter SRE

#510

Post by leftcenterright »

mtejo wrote: post #2

Hey, sorry for the new account, i just like to try my best to keep my online identity separate. this for better or for worse has my real name on it. Hope this is interesting!

Kudos for nice work!

What did you make of Mudge's report regarding resiliency of data-centers?

> Insufficient data center redundancy, 59 without a plan to cold-boot or recover from even minor overlapping data center failure, raising the risk of a brief outage to that of a catastrophic and existential risk for Twitter's survival.

- https://techpolicy.press/wp-content/uploads/2022/08/whistleb...

Post reply on HN