Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

681–690 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#681

Post by nashashmi »

coldtea wrote: post #535
oxfordmale wrote: post #443

Earlier quoted context omitted.

Let's do a thought experiment and see what functions aren't needed to keep the light on for 30 days: 1) HR 2) Legal 3) Sales 4) Marketing 5) Payroll 6) Admin staff 7) Most of Engineering, other than the bare minimum of L1/L2/3 support. As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend.

> As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend. On the other hand, a car without a second and third steering wheel, 20 windscreen wipers, and an oven in the back, keeps running just fine, even after the first bend...

Unless the first steering wheel is prone to breaking. Redundancy is the difference between a million dollar company and billion dollar company.

Of course getting workers to come on on weekends and midnight hours accomplishes the same thing.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#682

Post by bushbaba »

Yoric wrote: post #663
holoduke wrote: post #506

Earlier quoted context omitted.

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

Not tech but I'm convinced that just answering LEO requests from all over the world takes at least 500 people. That's without their managers, payrolls, etc. Likewise, bringing in Ad money would be a few more hundreds, because you need to chase leads in all countries. Getting the Ads to work? That's tech and I'd be surprised if it was less than 100 people, too.

Tracking ads conversions, targeting, refinement of models, latency reduction , data analytics, ads sales, sales ops, etc. yeah easily 1000 employees all in.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#683

Post by [deleted] »

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

[deleted]

Re: Why Twitter didn’t go down: From a real Twitter SRE

#684

Post by throwawaylinux »

fatneckbeardz wrote: post #650
throwawaylinux wrote: post #109

Earlier quoted context omitted.

I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.

Quoted post unavailable.

Lots of people are puritans, I was talking about the puritans making themselves terribly offended by that meme. If Musk is one too it doesn't mean they aren't, it's not based on some relative scale here. But I am using the lowercase puritan which probably doesn't fit him.

I didn't know that he lead an extremist cult though, not that I have ever cared to find out much about his life or beliefs. It wouldn't surprise me, I think a lot of the ruling class and ultra rich are terrible people who we should not be living under. That said I have noticed the claims of Musk's terribleness and extremism may have a tiny correlation with his unwillingness to toe the Democrat party line. Now quite possibly that's just a coincidence and there's absolutely no causation there, nevertheless if I were ever in a situation where that information would be important to a decision, it would behoove me to take it on its merits rather than on faith.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#685

Post by jq-r »

867-5309 wrote: post #537
yunohn wrote: post #444

Earlier quoted context omitted.

> I mean wholly crap, this reminds me of old German industry where people retire in the place they started. This is honestly uncalled for. Job hopping every 2-3yrs should not be an expected task.

is that down to the employer or employee?

Could be both.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#686

Post by twelve40 »

enkid wrote: post #502
tempestn wrote: post #452

Earlier quoted context omitted.

I think you're misinterpreting the comment you're replying to. They would agree with you that the tiny SRE team described in the article sounds very effective, and likely have a lot to do with why the site is still up and running currently. Work like that should continue. But if 1-3 people can have that degree of impact, what are the other 8000 doing? (Again, this is just me attempting to interpret the point made by…

So 1-3 people have a big impact, the other 7997 must not be doing anything? I don't think that logic follows.

it doesn't follow. The article posits that "many people think twitter headcount was bloated" then proceeds to describe a (presumably) really efficient work of a small SRE team. These two parts seem completely disconnected from each other - neither one proves, disproves or follows from the other - so it's unclear why the former was mentioned at all.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#687

Post by itsoktocry »

_3u10 wrote: post #587
wikfwikf wrote: post #553

Earlier quoted context omitted.

data architecture is bloat? Have you implemented a system which stores hundreds of billions of pieces of media content and makes different slices of them immediately available to hundreds of millions of users?

No but the people who made clickhouse did. I don’t think 1000s of engineers were required for that. Where’s the 1000s of engineers for Postgres? Most stuff that works is made by a handful of people. Look at io_uring it’s basically one guy at Facebook…

>Where’s the 1000s of engineers for Postgres?

You're comparing Postgres to Twitter?

If some of the people making comments like this actually work in tech, then yeah, maybe there is a lot of bloat to be cut.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#688

Post by marstall »

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

I'll add one: when usage scales beyond anticipated levels. then that code that is "good enough" will no longer be, and serious intervention may be required - by senior engineers with history.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#689

Post by WJW »

threeseed wrote: post #666
holoduke wrote: post #506

Earlier quoted context omitted.

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

> I am convinced you can run the entire tech stack with a team of a 100 people Please tell us in detail about the Twitter stack. Because I always find it fascinating how people think they can estimate the effort to maintain it whilst having next to no understanding what so ever of the tech stack.

I assume the reason for statements like that goes something like:

1. A single person can run a mastodon instance in their spare time. Spinning up some containers for the app, a background worker and a database is quite simple.

2. Modern devops tooling makes it fairly trivial to spin up 10k instances of a container instead of 1, by just altering a number in a k8s manifest somewhere.

3. Ergo, a single person equipped with modern tooling (and sufficient funding) could spin up any number of mastodon instances.

4. Twitter is just a big mastodon instance.

5. Now that keeping everything up is sorted, add another 99 devs for feature development and you are done.

Now this is obviously faulty logic because points 3 and 4 are very false, but they look reasonable enough at first glance.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#690

Post by itsoktocry »

carabiner wrote: post #477
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Yeah it's dancing around the question: Was Musk right? All signs so far are pointing to, yes . MBA's will be studying this for years.

>Was Musk right? All signs so far are pointing to, yes.

Huh? He's been in charge for, like, two weeks. Did you think it could implode the instant the engineers received pink slips? Let's give it a year before we say he was right.

Post reply on HN