Why Twitter didn’t go down: From a real Twitter SRE • Hacker News Board

Re: Why Twitter didn’t go down: From a real Twitter SRE

#851

Post by mattbrewsbytes » Tue, Nov 22, 2022, 2:34 PM UTC

redeyedtreefrog wrote: post #837

SpaceX and Tesla made affordable, reusable rockets, and affordable, high performance electric cars with a long range. Those things have obvious value, but required solving big engineering challenges to make them a reality. Musk doesn't seem to have any clear goal with twitter. There are vague ideas about making it subscription-funded rather than advertising-funded, and about having less censorship, but neither are th…

> Or that Musk is a genius who will make some sort of amazing Twitter 2.0

Based on texts exposed via the lawsuit about the purchase, I don't think this is the case. I don't think he, or his advisors, understand that the product at twitter (and other social media) is content moderation. You can have a vision of whatever type of content or pricing scheme you want but without solid content moderation you will lose advertisers, gain lawsuits (people were posting movies the other day) and lose users because the "feed" becomes a muddied mess. Users are only really the product when you can moderate their content to have profit via advertisers.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#852

Post by P5fRxh5kUvp2th » Tue, Nov 22, 2022, 2:35 PM UTC

threeseed wrote: post #666

holoduke wrote: post #506

Earlier quoted context omitted.

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

> I am convinced you can run the entire tech stack with a team of a 100 people Please tell us in detail about the Twitter stack. Because I always find it fascinating how people think they can estimate the effort to maintain it whilst having next to no understanding what so ever of the tech stack.

When people say this it's because they assume there's not stupid ass politics or moat building.

THEY could probably do it with 100 people, YOU cannot.

100 people is most likely within the ballpark for a group of people whose sole purpose is to write and maintain twitter's tech stack. Unfortunately, that is not NEARLY the sole purpose of most people in businesses and that adds all kinds of productivity hits.

What happens is that people like yourself become convinced that's the only way to operate.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#853

Post by donkeyd » Tue, Nov 22, 2022, 2:39 PM UTC

avereveard wrote: post #764

fnord123 wrote: post #472

Earlier quoted context omitted.

Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?

7493 given the article mentions the sre team being a 3 man operation

SRE for the Cache Team. I feel people are overlooking that part.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#854

Post by bdcravens » Tue, Nov 22, 2022, 2:40 PM UTC

I wonder if the Twitter situation may be the event that causes the industry to push back on large technical organization sizes.

I wonder how many reading this are starting to feel nervous about their own roles.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#855

Post by blisterpeanuts » Tue, Nov 22, 2022, 2:41 PM UTC

HyperSane wrote: post #214

zxcvbn4038 wrote: post #150

Now that Elon has cleaned house, when is Twitter hiring again? It might be a good opportunity to get in on the new ground floor.

Quoted post unavailable.

Is this based on personal experience, or are you going by some well known publicly available stories about Musk?

I've heard pretty much every possible opinion about the guy, but the ones that impress me the most are that he loves to go deep into technical topics with people who know their stuff, regardless of their credentials. That, to me, is a sign of a man with deep humility and desire to learn.

At the same time, he's not considered a very diplomatic person and evidently is rather hard to get along with both in his personal life and professionally. Yet, his accomplishments are substantial.

So I would go with maverick, different thinker, fk your feelings, kind of guy who not only isn't afraid to break a few eggs to make an omelette, but probably isn't even aware he's breaking the eggs :)

Re: Why Twitter didn’t go down: From a real Twitter SRE

#856

Post by tiahura » Tue, Nov 22, 2022, 2:41 PM UTC

Yoric wrote: post #663

holoduke wrote: post #506

Earlier quoted context omitted.

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

Not tech but I'm convinced that just answering LEO requests from all over the world takes at least 500 people. That's without their managers, payrolls, etc. Likewise, bringing in Ad money would be a few more hundreds, because you need to chase leads in all countries. Getting the Ads to work? That's tech and I'd be surprised if it was less than 100 people, too.

Not tech but I'm convinced that just answering LEO requests from all over the world takes at least 500 people.

I subpoena telcos all the time. My sense is that the number is closer to 2 to 3 dozen.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#857

Post by BeefWellington » Tue, Nov 22, 2022, 2:42 PM UTC

cdf wrote: post #564

holoduke wrote: post #506

Earlier quoted context omitted.

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

Getting it to about 90% working in a single country or just the general western sphere, probably 100 will suffice. Making it globally available and legally compliant, that's where the next few thousand folks come in.

Indeed. Just translating it could be a team of at least N+0.5 people, where N is the number of languages you want to translate your product to (the +0.5 accounting for translation verifications). Even if you outsource it that just slows down feature deployment IME.

The people shouting loudly about how Twitter must have been so bloated are really just shouting their obvious inexperience working at global scales or their localized ambitions.

Could there be too many employees at Twitter? Sure. Most companies have dead weight. The number who were "extra" is probably not 9/10ths the employees though.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#858

Post by lightbendover » Tue, Nov 22, 2022, 2:42 PM UTC

_3u10 wrote: post #587

wikfwikf wrote: post #553

Earlier quoted context omitted.

data architecture is bloat? Have you implemented a system which stores hundreds of billions of pieces of media content and makes different slices of them immediately available to hundreds of millions of users?

No but the people who made clickhouse did. I don’t think 1000s of engineers were required for that. Where’s the 1000s of engineers for Postgres? Most stuff that works is made by a handful of people. Look at io_uring it’s basically one guy at Facebook…

I am entirely missing how Clickhouse solves the problem Twitter is solving out of the box, do you care to explain? Sub-question: how does Clickhouse provide monetization for a product?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#859

Post by nemo44x » Tue, Nov 22, 2022, 2:43 PM UTC

fnord123 wrote: post #472

donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?

I've heard stories of teams that were more or less dedicated to nothing useful except policing the company, product, and users for woke violations. All these "affinity groups" to ensure voices were "being heard" when decisions were being made or some garbage like that. In essence, at some point there was a takeover and senior leadership either agreed with it or simply didn't want to get into trouble like Google did with these things, so they just went along with it all.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#860

Post by blisterpeanuts » Tue, Nov 22, 2022, 2:44 PM UTC

justahuman74 wrote: post #392

hot_gril wrote: post #303

You can usually do more with less if the remaining few are up to the challenge. It will come at some cost, like having to "move fast and break things" more. There can even be a positive side if the previous situation had a lot of friction due to "too many cooks," as I've experienced on several teams. What's more worrisome is the alleged 80hr work weeks. Yes, two SWEs on a team aren't as good as one SWE working double…

> cut corners on actually verifying the subscribers Did they actually verify anything other than the ability to pay $8? It seems wild to me that they thought it would work out just fine

Feels like one of those "ideas that sounded good in the conference room" that turns out to have unintended consequences.

Hopefully they can fix this and have an effective way to verify identity... or else just give it up and go back to the old, creaky, somewhat effective method since people are used to it.