Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

821–830 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#821

Post by adam_arthur »

pclmulqdq wrote: post #808
adam_arthur wrote: post #759

Earlier quoted context omitted.

Well, WhatsApp had ~50 employees and Instagram around ~15 when FB acquired them, and they were around the same order of magnitude of complexity as Twitter. The only concern Id have is that by having so many people, your design probably comes to rely on them whereas a smaller team would be forced to make the system easier to maintain. Personally, if I were Elon, I’d build an entirely new backend and point the clients…

A lot of people who don't understand why Twitter owns two datacenters point to "complexity" as an argument and completely disregard scale. It turns out that massive scale adds a lot of complexity to a system, particularly around many-to-many pubsub systems (like social media). It also means a lot of features, like compliance with government regulations around the world.

WhatsApp had a Billion users and 50 employees. You can say that Twitter is incrementally more complex due to wider broadcasting, but it’s also 1/5 the user count.

Everybody stating this number of employees is necessary to maintain a Twitter scale system is simply wrong.

And as technology progresses, fewer and fewer people are needed to maintain the same size system

Many just don’t want to believe that the leverage in employees favor in the tech sector is fading fast, and their labor is not going to warrant 500k comp with marginal effort anymore

Re: Why Twitter didn’t go down: From a real Twitter SRE

#822

Post by vonmoltke »

__warlord__ wrote: post #217
wyclif wrote: post #114

Earlier quoted context omitted.

I was surprised to hear that a cutting edge social media app uses Mesos. Why did they choose that over other options?

IIRC Mesos was an internal tool (at Twiiter) that got released under the Apache umbrella later on. Then Mesosphere (a company) wanted to bring it to the enterprise market but at the time was competing with Kubernetes... and we all know who won.

Aurora, a container manager on top of Mesos, was developed in-house. Mesos itself was not.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#823

Post by blueprint »

joenot443 wrote: post #763
matwood wrote: post #752

Earlier quoted context omitted.

I agree those were odd takes. I've likened firing most of the engineers to taking your hands off the wheel in the car. It won't crash immediately, but it doesn't mean the car can go driverless. With that said, there are differences between internal systems and something like Twitter on the public internet. I assume that Twitter is a system under constant attack. What happens when the next log4shell level vulnerabilit…

If Twitter went another month without an outage, how would you adjust your opinion? How about a year? The car analogy is amusing, but how much does it really hold up? Have we ever seen another major social media company drop this much of its staff in one go? I certainly can’t think of an example. I think we’re in somewhat uncharted waters here. A driverless car won’t last long, we know that for a fact. I think it rem…

yes, maybe a car without mechanics rather than without a driver. drivers can be really bad, too. so bad that they forget to steer or steer badly, etc. but the steering wheel is categorically the interface made for non-engineer usage.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#824

Post by blueprint »

hef19898 wrote: post #726
Aeolun wrote: post #719

Earlier quoted context omitted.

> thought that Twitter would crash within days of the engineers being fired Because they work for companies where the product would fail within days of them being fired themselves.

That's what people keep telling themselves. Truth is, everybody is replaceable. And usually, loss of institutional knowledge takes a while to show real effects.

how can you possibly say everyone is replaceable while next giving an example of a case where it's not only not true but potentially so invisible in its consequences as to make it hard or impossible to satisfactorily replace someone? a "person" can be rebuilt but not replaced. companies and societies aren't actually bose-einstein condensates. everyone is replaceable is just what different types of people say as a coping mechanism. it's a disgusting thing to promulgate too. maybe if someone is equivalent to a cog they can be replaced but humans are not exactly standardized cogs. what you mean to say, and may get in the habit of repeating instead, is that we don't necessarily need a specific person even though we are accustomed to them. it still might take you 10 years to find someone who does 80% of the same things as the other person while the missing 20% is what made that person such a unique hire in the first place. i find it a disgusting phrase that is so typical of the view of a society like ours today that is losing view of the value of life.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#825

Post by awithrow »

shapefrog wrote: post #629
mjr00 wrote: post #451

Earlier quoted context omitted.

It's crazy to think about, but many people who use and build software today, including HN readers/commenters, are young enough to have only been exposed to the SaaS, cloud-first era, where software built with microservices deployed from CI/CD systems multiple times per day is just the way things are done. You're totally right; if you don't make changes to the software, it's unlikely to spontaneously stop working, esp…

> The cloud/SaaS approach You dont have to push constant updates for a cloud / SaaS product - many chose to - but ultimately you dont have to. A year of 'no new features' should be something customers and vendor alike benefit from.

Depending on your stack, security patching can constitute a non-trivial amount of changes as well.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#826

Post by vasergen »

mgkimsal wrote: post #589
mjr00 wrote: post #451

Earlier quoted context omitted.

It's crazy to think about, but many people who use and build software today, including HN readers/commenters, are young enough to have only been exposed to the SaaS, cloud-first era, where software built with microservices deployed from CI/CD systems multiple times per day is just the way things are done. You're totally right; if you don't make changes to the software, it's unlikely to spontaneously stop working, esp…

The other side of that is browsers. Even if you don’t change your code, the platform people are running your code in changes, automatically in many cases. New JS or CSS behavior in next safari or chrome? You need to patch/push to accommodate running environments that are outside your control.

JS in fact has very good backwards compatability, this is by the way the reason why "old stuff" are not removed and are still in the language.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#827

Post by overthemoon »

h11h wrote: post #493
minhazm wrote: post #373

Earlier quoted context omitted.

Many people who have worked with Musk have shared similar sentiments in interviews. But it seems that people just refuse to believe any of it. People think that there's no way it's possible for someone to be that deeply technical and be a CEO of multiple companies at the same time. I've talked to people about it and they straight up refuse to believe it saying that it's impossible and that any evidence of him being t…

I'm inclined to believe this is true. The problem is that Twitter's challenges are social/political, not technical, and Musk has demonstrated little competence in this area.

This is the absolute root of what's going on, right here. Twitter is only nominally a tech company; it's a media company. It may be that he had to cut the fat over there, I don't take issue with that necessarily (though I certainly do take issue with the disrespectful way he went about it), but image is incredibly important at a media company and he's notoriously bad at comms except with a small subset of people. Twitter needs sensible policy and thoughtful communication, and he wants to ram his ideology through it like he would shake up any technical process.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#828

Post by water8 »

donkeyd wrote: post #372
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

No post body was provided.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#829

Post by water8 »

jameshart wrote: post #755
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

That small team seems to have been running the caches for other teams, by using infrastructure provided by another team, in two massive datacenters operated by other teams, using monitoring tools managed by another team, and a ticketing system run by another, on hardware purchases by another team… All just to put caching in front of services that actually do anything.

Can’t forget about those people that ordered that hardware once. He probably had to go to the business website and everything.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#830

Post by donkeyd »

throw0101c wrote: post #744
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

> In a start-up many of these roles can be ignored, becaus growth > stability. In a large organization, part of the bloat helps insure a certain amount of stability that's necessary to keep an organization alive. It also (a) increases the bus factor, [1] and (b) allows people to take vacations and time off without having to watch their phones like hawk. [1] https://en.wikipedia.org/wiki/Bus_factor

Good point. I know this as the Mack Truck Theory. For the project I'm working on right now, there's a couple of incredibly valuable people that would cause a pretty significant issue if they disappeared.
Post reply on HN