Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Production Twitter on one machine? 100Gbps NICs and NVMe are fast

thume.ca

Post reply on HN
Print view View original

181–190 of 500 posts

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#181

Post by Xeoncross »

BeefWellington wrote: post #40

I'm going to preface this criticism by saying that I think exercises like this are fun in an architectural/prototyping code-golf kinda way. However, I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount. e.g.: Quote tweets do not go against the size limit of the tweet field at Twitter. Likely they are embedding a tweet reference in some manner or ot…

> I think the author critically under-guesses the sizes of things (even just for storage) by a reasonably substantial amount.

While true, and not to take away from the parent comment, I've noticed that the size of things is often partially the result of scaling out horizontally. Most companies I've worked at end up with a lot of duplicate records as each subsystem might want a copy or to cache a copy.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#182

Post by SilverBirch »

reacharavindh wrote: post #117
SilverBirch wrote: post #109

I think one of the under-estimated interesting points of twitter as a business is that this is the core. Yes, Twitter is 140 characters, it's got "300m users" which is probably 5m real heavy users. So yes, you could do a lot of "140 characters, a few tweets per person, few million users" on very little hardware. But that's why Twitters a shit business! How much RAM did your advertising network need? Becuase that is w…

The author claimed early on, and very clearly that this was a fun exercise of thought and engineering rather than saying “Look this is how Twitter should be run”. After all this is Hacker News. Such exercises, and engaging other hackers to pick something out of there is how we progress(and get our tickles). So, may be instead think about how one could tackle the advertising/indexing needs in a similar fashion(could i…

Yeah I completely get that, but I think a lot of hacker news tends to think of companies as the sum of their engineering resources, rather than what they really are. Which is weird, because it's hosted by YC which is meant to be the polar opposite of that. The point of my comment was that the decision making that lead to Twitter's design actually make sense when you understand the business model behind them. It's not good enough to just run a webpage.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#183

Post by icedchai »

lelandfe wrote: post #174
mr90210 wrote: post #115

Earlier quoted context omitted.

[flagged]

If an inspector reviews your house and finds no issues, that is indeed evidence of absence.

Not necessarily. The inspector could be corrupt, bad at their job, whatever.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#184

Post by threeseed »

seritools wrote: post #176
threeseed wrote: post #132

Earlier quoted context omitted.

> just shuffling tweet sized buffers to network offload cards Except that's not what it is doing at all. It assembles all the Tweets internally, applies an ML model to produce a finalised response to the user.

> if it really was [which it isn't]

But many people including the OP think it is.

It’s like me running a web crawler on my phone and saying I can replace Google.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#185

Post by mcenedella »

sitkack wrote: post #180

I am both embarrassed and disappointed with the negativity this post has attracted. A litanny of "gotchas", where someone attempts to best the OP. What about x, y and z? It can't possibly scale. Twitter is so much more than this, etc. The OP isn't making the assertion that Twitter should replace their current system with a single large machine. The whole thread paints a picture of HN like it is full of a bunch of hal…

Amen!

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#186

Post by morphle »

mpoteat wrote: post #58
morphle wrote: post #22

Why not a single FPGA with 100Gbps ethernet or pcie with NVM attached? Around $5K for the hardware and $5K for the traffic per month. The software would be a bit trickier to write, but you now get 100x performance for the same price

Let's spend multi-million dollars a year on a team of highly specialized FPGA engineers writing assembly and HDL so that we can save 5k a month. Feature velocity will be 100x slower as well, but at least our application is efficient. I think that this may make sense for some applications, but I also think that if you can utilize software abstractions to improve developer efficiency, it reduces risk in the long run.

Those millions of dollars have already been spent. For example the P4 [1] language (a HDL language) and the Tofino 3 chip. It started out as FPGA (NetFPGA) to do programmable packet routing at linespeed. You now have the P4 language to define your packet routers with and generate your code. Ten years later we have 25 Tbps software defined packet routers.

[1] https://en.wikipedia.org/wiki/P4_(programming_language)

[2] https://www.intel.com/content/www/us/en/products/network-io/...

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#187

Post by Spooky23 »

varunkmohan wrote: post #119
PragmaticPulp wrote: post #65

Earlier quoted context omitted.

> However, it does show how much real compute bloat these companies actually have No, it doesn’t. It’s a fun exercise in approaching Twitter as an academic exercise. It ignores all of the real-world functionality that makes it a business rather than a toy. A lot of complicated businesses are easy to prototype out if you discard all requirements other than the core feature. In the real world, more engineering work oft…

Genuinely asking, why do you think Twitter needs 24 million vcpus to run? This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their reve…

Because at that time, Whatsapp didn’t do all of the stuff that you need to do. No lawful intercept, no abuse detection/prevention, etc.

I don’t know enough about Twitter to assess their infrastructure, but I know that it easy to run lean until there’s a problem, and then you get trapped.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#188

Post by Xeoncross »

sitkack wrote: post #180

I am both embarrassed and disappointed with the negativity this post has attracted. A litanny of "gotchas", where someone attempts to best the OP. What about x, y and z? It can't possibly scale. Twitter is so much more than this, etc. The OP isn't making the assertion that Twitter should replace their current system with a single large machine. The whole thread paints a picture of HN like it is full of a bunch of hal…

I must admit, as a bystander I'm torn here.

I love that Tristan put out this post and made it so detailed with plenty of assumptions to cover. I also like to hear about possible issues and assumptions which the crowd calls out. Even naysayers can be helpful.

I want both, but I don't want to crowd to go to far and kill the desire to produce this kind of content.

Re: Production Twitter on one machine? 100Gbps NICs and NVMe are fast

#189

Post by knubie »

> I’m not sure how real Twitter works but I think based on Elon’s whiteboard photo and some tweets I’ve seen by Twitter (ex-)employees it seems to be mostly the first approach using fast custom caches/databases and maybe parallelization to make the merge retrievals fast enough.

I think Twitter does (or at some point did) use a combination of the first and second approach. The vast majority of tweets used the first approach, but tweets from accounts with a certain threshold of followers used the second approach.

Post reply on HN