Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

751–760 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#751

Post by jillesvangurp »

gjulianm wrote: post #694
jillesvangurp wrote: post #514

Earlier quoted context omitted.

You underestimate Elon Musk. Many people have done that before and lost that bet. If anything, he repeatedly succeeded in building world class software and hardware teams for Tesla, SpaceX, and a few other companies. The notion that he won't be able to attract world class talent is ludicrous. Yes, he is a bit of a liability and his management style is obnoxious and unconventional. But he does get things right once in…

> If anything, he repeatedly succeeded in building world class software and hardware teams for Tesla, SpaceX, and a few other companies The point is that Twitter doesn't really needed someone to build a world class software and hardware team. The technical challenges in reliability and speed seemed pretty much solved or on track to be solved already. The problem of Twitter was that they never knew how to properly man…

Exactly. it had a team problem. I think it's safe to use past tense now because that team is mostly gone now. It still has some team challenges but those he can fix with strategic hires and hard work.

Fixing the community starts with rolling back all the things that clearly did not work. He's using the sledge hammer method there too. So, not very subtle but generally just getting of rid of a lot of failed and failing policy.

The technical challenges in speed and scaling are not challenges at all anymore. Twitter built a lot of stuff in house when you couldn't get that stuff as a commodity. That has changed since then. You need a cache, you can get one from any number of cloud providers or spin up something off the shelf you run yourself. Same with databases, CDNs, large scale object storage, search infrastructure, message brokers, and all the rest. So, yes, there might be a need for changing some of that necessitated by some key people disappearing but it's not a massive technical challenge.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#752

Post by matwood »

itsoktocry wrote: post #676
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

> removing engineers won't instantly crash the product. It'll happen slowly It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.

I agree those were odd takes. I've likened firing most of the engineers to taking your hands off the wheel in the car. It won't crash immediately, but it doesn't mean the car can go driverless.

With that said, there are differences between internal systems and something like Twitter on the public internet. I assume that Twitter is a system under constant attack. What happens when the next log4shell level vulnerability comes out?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#753

Post by mk_stjames »

wankle wrote: post #30

I can't help but feel it might not be legal to post all those details.

Out of all the comments here, I can't believe this is the only one that shares my sentiment. I'm not a software engineer, but in my area of engineering if I ever revealed anything publicly as detailed as this kind of architecture; for instance, the failure modes effect analysis framework use at a certain aerospace company... or details of the simulation models involving the crash analysis at a certain car company, so soon after departing one of those companies, I'd expect a letter from a lawyer within a week. In most places I've worked I don't think I could even post anything on the internet invoking the name of the company unless it was a repost of some public information the company itself had previously released. Do the vast majority of people here on HN/in Silicon Valley not have 5-year NDA's? Or do they and they just not care?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#754

Post by vonmoltke »

threeseed wrote: post #435
intelVISA wrote: post #426

Earlier quoted context omitted.

> certs won't get renewed because institutional knowledge has left the door. The parallel reality where you need to be a veteran SRE with an MIT degree to operate the arcane tool 'certbot'.

They aren't talking about the front-end certificates which expire in Feb 2023. It's likely the ones to encrypt all of the traffic involving the Finagle micro-services, data sources, observability systems etc. And I suspect the issue there is that you are going to need to do a rolling restart. Which I personally would not want to be doing if 90% of the company is no longer there.

The way TLS was integrated into Finagle, most services should not need to be restarted to pick up and use their new certs. That said, there are certain core services that will require manual intervention, and there will inevitably be some services that should auto-update but do not.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#755

Post by jameshart »

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

That small team seems to have been running the caches for other teams, by using infrastructure provided by another team, in two massive datacenters operated by other teams, using monitoring tools managed by another team, and a ticketing system run by another, on hardware purchases by another team…

All just to put caching in front of services that actually do anything.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#756

Post by anon400232 »

kortilla wrote: post #565
mschuster91 wrote: post #517

Earlier quoted context omitted.

> I mean wholly crap, this reminds me of old German industry where people retire in the place they started. German here. I think this actually is a huge part of the success of the famous Mittelstand - all that institutional knowledge these people have is extremely valuable. It's not just basic stuff like "know time tracking, billing and other admin systems and internal processes", but also the stuff that really can s…

> the problem is US-centric capitalism cannot quantify that impact (and it doesn't want to, given that attrition-related problems are long-term issues with years of time to impact) This is false. People invest on many year horizons all of the time when they believe in the company. If there was good evidence that having an average tenure of 5+ years was a great boost to the company, people would clamor to invest in co…

> This is false. People invest on many year horizons all of the time when they believe in the company.

Granted, but they usually have the option to sell at many points along the way.

Think about an alternative financial instrument. What if you could invest $X in a public company (and get a higher rate of return because of the higher risk) but you lose the ability to sell until five years have passed?

This is one kind of option that might be worth trying. I want to see more mechanisms promoting long-term investment in our public markets. (Bonds are not identical to what I described BTW.)

The lack of these kinds of options (sure, bonds exist but I've not noticed them used with any notable significance in public companies) is why people talk about the short term quarterly focus of Wall Street.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#757

Post by matwood »

holoduke wrote: post #506
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

For US only? This sounds like the comment complaining about app size when an Uber engineer showed up and talked about handling 100s of methods of worldwide payments. Many in the US seem to think the internet only exist in the US, and that US rules apply everywhere.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#758

Post by forgetfulness »

hef19898 wrote: post #726
Aeolun wrote: post #719

Earlier quoted context omitted.

> thought that Twitter would crash within days of the engineers being fired Because they work for companies where the product would fail within days of them being fired themselves.

That's what people keep telling themselves. Truth is, everybody is replaceable. And usually, loss of institutional knowledge takes a while to show real effects.

It depends on how well or poorly run the business is.

I worked at a company where everything hinged solely on one guy working from another country. When he left, loss of institutional knowledge took about three days to show real effects as things also came down crashing.

I worked hard to make _myself_ replaceable for when I left, it was a pretty good exercise, but me having that degree of freedom was symptomatic of the problems of the company.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#759

Post by adam_arthur »

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Well, WhatsApp had ~50 employees and Instagram around ~15 when FB acquired them, and they were around the same order of magnitude of complexity as Twitter.

The only concern Id have is that by having so many people, your design probably comes to rely on them whereas a smaller team would be forced to make the system easier to maintain.

Personally, if I were Elon, I’d build an entirely new backend and point the clients to that rather than trying to incrementally improve what they have.

Get 50-100 10x engineers that are loyal to Elon, with big equity stakes, and crush it

Re: Why Twitter didn’t go down: From a real Twitter SRE

#760

Post by gjulianm »

jillesvangurp wrote: post #751
gjulianm wrote: post #694

Earlier quoted context omitted.

> If anything, he repeatedly succeeded in building world class software and hardware teams for Tesla, SpaceX, and a few other companies The point is that Twitter doesn't really needed someone to build a world class software and hardware team. The technical challenges in reliability and speed seemed pretty much solved or on track to be solved already. The problem of Twitter was that they never knew how to properly man…

Exactly. it had a team problem. I think it's safe to use past tense now because that team is mostly gone now. It still has some team challenges but those he can fix with strategic hires and hard work. Fixing the community starts with rolling back all the things that clearly did not work. He's using the sledge hammer method there too. So, not very subtle but generally just getting of rid of a lot of failed and failing…

> Fixing the community starts with rolling back all the things that clearly did not work.

And which ones are those? Knowing what did and did not work is an actual challenge by itself.

> You need a cache, you can get one from any number of cloud providers or spin up something off the shelf you run yourself. Same with databases, CDNs, large scale object storage, search infrastructure, message brokers, and all the rest. So, yes, there might be a need for changing some of that necessitated by some key people disappearing but it's not a massive technical challenge.

The massive technical challenge is migrating existing infrastructure to something off the shelf, then finding and fixing the new bugs in that existing infrastructure and/or your deployment/configuration. That shouldn't be underestimated.

Post reply on HN