Production Twitter on one machine? 100Gbps NICs and NVMe are fast • Hacker News Board

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#161

Post by sayrer » Sat, Jan 07, 2023, 10:46 PM UTC

sayrer wrote: post #137

trishume wrote: post #121

Earlier quoted context omitted.

I think I'm pretty careful to say that this is a simplified version of Twitter. Of the features you list: - spam detection: I agree this is a reasonably core feature and a good point. I think you could fit something here but you'd have to architect your entire spam detection approach around being able to fit, which is a pretty tricky constraint and probably would make it perform worse than a less constrained solution…

"web previews: I'd do this by making it the client's responsibility." Actually a good example of how difficult the problem is. A very common attack is to switch a bit.ly link or something like that to a malicious destination. You would also DoS the hosts... as the Mastodon folks are discovering ( https://www.jwz.org/blog/2022/11/mastodon-stampede/ ) For blocks/mutes, you have to account for retweets and quotes, it's…

Here are some pointers:

"Our approach to blocking links" https://help.twitter.com/en/safety-and-security/phishing-spa...

"The Infrastructure Behind Twitter: Scale" https://blog.twitter.com/engineering/en_us/topics/infrastruc...

"Mux" https://twitter.github.io/finagle/guide/Protocols.html#mux

I do agree that some of this could be done better a decade later (like, using Rust for some things instead of Scala), but it was all considered. A single machine is a fun thing to think about, but not close to realistic. CPU time was not usually the concern in designing these systems.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#162

Post by jiggawatts » Sat, Jan 07, 2023, 10:47 PM UTC

Something I've found a lot modern IT architects seem to ignore is "write amplification" or the equivalent effect for reads.

If you have a 1 KB piece of data that you need to send to a customer, ideally that should require less than 1 KB of actual NIC traffic thanks to HTTP compression.

If processing that 1 KB takes more than 1 KB of total NIC traffic within and out of your data centre, the you have some level of amplification.

Now, for writes, this is often unavoidable because redundancy is pretty much mandatory for availability. Whenever there's a transaction, an amplification factor of 2-3x is assumed for replication, mirroring, or whatever.

For reads, good indexing and data structures within a few large boxes (like in the article) can reduce the amplification to just 2-3x as well. The request will likely need to go through a load balancer of some sort, which amplifies it, but that's it.

So if you need to process, say, 10 Gbps of egress traffic, you need a total of something like 30 Gbps at least, but 50 Gbps for availability and handling of peaks.

What happens in places like Twitter is that they go crazy with the microservices. Every service, every load balancer, every firewall, proxy, envoy, NAT, firewall, and gateway adds to the multiplication factor. Typical Kubernetes or similar setups will have a minimum NIC data amplification of 10x on top of the 2-3x required for replication.

Now multiply that by the crazy inefficient JSON-based protocols, the GraphQL, an the other insanity layered on to "modern" development practices.

This is how you end up serving 10 Gbps of egress traffic with terabits of internal communications. This is how Twitter apparently "needs" 24 million vCPUs to host text chat.

Oh, sorry... text chat with the occasional postage-stamp-sized, potato quality static JPG image.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#163

Post by [deleted] » Sat, Jan 07, 2023, 10:47 PM UTC

d23 wrote: post #145

varunkmohan wrote: post #119

Earlier quoted context omitted.

Genuinely asking, why do you think Twitter needs 24 million vcpus to run? This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their reve…

Presumably there's an entire data engineering / event processing pipeline that's being used to track user interactions at a fine grained level. These events are going to be aggregated and munged by various teams for things like business analytics, product / experiment feature analysis, ad analysis, as well as machine learning model feature development (just to name a few massive ones off the top of my head). Each of…

[deleted]

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#164

Post by veec_cas_tant » Sat, Jan 07, 2023, 10:47 PM UTC

ricardobeat wrote: post #153

d23 wrote: post #145

Earlier quoted context omitted.

Presumably there's an entire data engineering / event processing pipeline that's being used to track user interactions at a fine grained level. These events are going to be aggregated and munged by various teams for things like business analytics, product / experiment feature analysis, ad analysis, as well as machine learning model feature development (just to name a few massive ones off the top of my head). Each of…

And for the most part, this herculean effort is wasted. Most people just want to see latest tweets from people they follow. Everything else is fluff to manipulate engagement metrics, pad resumes and attempt to turn twitter into something it's users never wanted.

Just guessing, but a lot of the resources are probably devoted to making money for the business, not padding resumes. Others have pointed it out, but showing tweets doesn't generate revenue without additional infrastructure.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#165

Post by Snoozus » Sat, Jan 07, 2023, 10:49 PM UTC

varunkmohan wrote: post #119

PragmaticPulp wrote: post #65

Earlier quoted context omitted.

> However, it does show how much real compute bloat these companies actually have No, it doesn’t. It’s a fun exercise in approaching Twitter as an academic exercise. It ignores all of the real-world functionality that makes it a business rather than a toy. A lot of complicated businesses are easy to prototype out if you discard all requirements other than the core feature. In the real world, more engineering work oft…

Genuinely asking, why do you think Twitter needs 24 million vcpus to run? This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their reve…

Because the actual product is not showing people tweets but to optimize who to show which ads based on their previous interactions with the site. This is many orders of magnitude harder.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#166

Post by danpalmer » Sat, Jan 07, 2023, 10:49 PM UTC

varunkmohan wrote: post #119

PragmaticPulp wrote: post #65

Earlier quoted context omitted.

> However, it does show how much real compute bloat these companies actually have No, it doesn’t. It’s a fun exercise in approaching Twitter as an academic exercise. It ignores all of the real-world functionality that makes it a business rather than a toy. A lot of complicated businesses are easy to prototype out if you discard all requirements other than the core feature. In the real world, more engineering work oft…

Genuinely asking, why do you think Twitter needs 24 million vcpus to run? This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their reve…

Being E2E encrypted, WhatsApp can’t do much with the content, so it is much closer to a naive bitshuffler than Twitter.

Twitter, while still not profitable (maybe it was in some recent quarters?) was much closer to it, having all the components necessary to form a reasonable ad business. For ads, analytics is critical, plus all the ad serving, plus it’s a totally different scale of compute being many to many rather than one to ~one.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#167

Post by sofixa » Sat, Jan 07, 2023, 10:50 PM UTC

ricardobeat wrote: post #153

d23 wrote: post #145

Earlier quoted context omitted.

Presumably there's an entire data engineering / event processing pipeline that's being used to track user interactions at a fine grained level. These events are going to be aggregated and munged by various teams for things like business analytics, product / experiment feature analysis, ad analysis, as well as machine learning model feature development (just to name a few massive ones off the top of my head). Each of…

And for the most part, this herculean effort is wasted. Most people just want to see latest tweets from people they follow. Everything else is fluff to manipulate engagement metrics, pad resumes and attempt to turn twitter into something it's users never wanted.

Most people probably follow more people than they're capable of reading all the latest tweets of, so some sort of ranking/prioritisation makes total sense. And Twitter is ad funded, so they need to also show relevant ads where it makes sense/money.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#168

Post by yodsanklai » Sat, Jan 07, 2023, 10:50 PM UTC

fleddr wrote: post #158

"Through intense digging I found a researcher who left a notebook public including tweet counts from many years of Twitter’s 10% sampled “Decahose” API and discovered the surprising fact that tweet rate today is around the same as or lower than 2013! Tweet rate peaked in 2014 and then declined before reaching new peaks in the pandemic. Elon recently tweeted the same 500M/day number which matches the Decahose notebook…

> Musk bought garbage for top dollar

Totally out of topic here, but could be he just wants the ability to amplify his own ideas. Also, why measure Twitter value (arbitrarily?) by number of unique tweets, rather than by read tweets?

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#169

Post by wmf » Sat, Jan 07, 2023, 10:51 PM UTC

jasonhansel wrote: post #19

If you really wanted to run Twitter on one machine at any cost, wouldn't an IBM mainframe be much more practical? You can even run Linux on them now. The specs he cites would actually be fairly small for a mainframe, which can reach up to 40TB of memory. I'm not saying this is a good idea, but it seems better than what the OP proposes.

No. A Genoa server is probably faster than a z16 and a Superdome Flex is definitely faster.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#170

Post by veec_cas_tant » Sat, Jan 07, 2023, 10:52 PM UTC

varunkmohan wrote: post #127

judge2020 wrote: post #33

Earlier quoted context omitted.

On the other hand, Twitter does (or did) handle over 450 million monthly active users (based on stats websites), with a target for 315 monetizable daily active users (based on their earnings calls pre-privatization). Handling that amount of concurrency and beaming millions of tweets a day to home feeds and notifications is going to be logistically hard.

Posted this on a comment above but systems like Whatsapp likely sent an insane amount of data as well but used only 16 servers over 1.5 billion users at time of acquisition. Modern NICs can handle millions of requests a second - I still feel there is a lot of excess here.

Feels like the comparison is irrelevant. I'm guessing WhatsApp would have infrastructure challenges if all of their chats were group messages including the entirety of their user base, search, moderation, ranking, ads, etc. Isn't WhatsApp more comparable to only DMs?